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Executive Summary 


The project 


The Ashford Teaching Alliance (ATA) Research Champion project (‘the programme’) was a pilot 
intervention aimed at developing teaching expertise and practice by promoting the use of educational 
research in decision-making and teacher practice. The programme ran for one academic year 
(2014/2015) in five schools within the ATA. Delivery was led by a ‘Research Champion’, a senior 
teacher based at one of the schools who worked with research leads, other teachers, and senior 
leaders to promote engagement with research evidence. The programme had four key components: 
‘audits’ of needs and research interests for individual schools; a series of research symposia for 
teachers; termly research and development ‘twilight forums’ (events held at the end of the school day 
at one of the participating schools); and bespoke research brokerage. 


The principal objective of this pilot study was to explore whether, and to what extent, research 
communication and engagement strategies had the potential to improve teachers’ use of, and 
attitudes towards, academic research to support pupils’ progress. The project was funded through the 
EEF Research Use in Schools grants round which supported studies to test ways of increasing the 
impact of educational research in schools. It was jointly funded by the EEF, the Department for 
Education and the Mayor’s London Schools Excellence Fund. 


Key Conclusions 


1. There was no evidence that teachers’ attitudes towards research, or their use of research 
evidence in teaching practice, changed during the intervention. 


2. Teachers found the research symposia and twilight events valuable, particularly as 
opportunities to learn about developments in educational research and reflect on teaching 
practice outside the classroom. 


3. Attendance and engagement in the programme was occasionally low due to time pressures 
faced by teachers. This posed a serious threat to the feasibility of the programme. 


4. A greater commitment from senior leadership teams to fully support staff with release time 
and classroom cover is likely to be necessary for successful implementation. 


5. The programme requires further development before it is ready for a trial. In particular it 
requires a clearer specification of the key features of the programme in terms of structure, 
content, and which components are required, and better information for schools on how 
much commitment is needed from teachers. 


What are the findings 


The programme largely ran as intended and was generally positively perceived by participating 
teachers. Those who attended the ‘twilight sessions’ and symposia found them valuable. However, 
while the structure of the programme and its outputs were as planned, levels of teacher engagement 
varied. Overall, therefore, the evidence on the programme's feasibility was mixed. 


The evaluation helped to highlight some of the practical barriers to engagement with, and 
implementation of, educational research. One of the main barriers was a lack of time to fully engage in 
the programme, which was related to competing priorities in schools and varying levels of buy-in from 
senior leadership teams. The findings suggest that systematic implementation of research evidence 
would require sustained engagement of senior leaders, through an enhanced process of mentoring 
and support. 


We believe that the Ashford Teaching Alliance Research Champion model is not currently ready to be 
evaluated in a trial. The programme requires a clearer specification of the key features of the 
programme in terms of structure (activities and outputs), content (topics covered at the training 
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events), and which components are required, and better information for schools on how much 
commitment is needed from teachers. 


The process evaluation identified some key learning points. Developers should: 


1. explore ways to ensure participating staff are given regular, dedicated time for the 
programme—in particular, release time to attend all events and to engage with the brokerage 
service, and time to plan, implement, and review changes in classroom practice; 

2. foster support from senior leaders at the school—encouraging buy-in from senior leadership 
teams would lead to more support for staff, including release time and classroom cover, as 
well as greater likelihood that learning from the project would be shared and taken forward 
across the whole school; 

3. allow flexibility for schools to tailor strategies to their own context—this was viewed as key to 
promoting engagement and buy-in from teachers and senior leadership teams; and 

4. provide practical examples and materials that could be used to facilitate classroom 
implementation, with a focus on simple strategies expected to bring ‘quick gains’. 


How was the evaluation of the pilot conducted? 


This was a mixed-methods (qualitative and quantitative) study with several iterative components. A 
model describing how the intervention would work in practice was drafted by NatCen researchers in 
consultation with the Research Champion following a workshop to identify the resources, activities, 
outputs, and intended outcomes of the programme. 


Teachers at the participating schools were surveyed at the beginning and end of the academic year. 
Quantitative data on teacher attitudes and behaviours was collected and analysed to identify any 
changes. The process evaluation was based on depth interviews, observations of training events, and 
the outcomes survey. Interviews and observations took place throughout the academic year in order to 
capture experiences and views of participants as the intervention progressed. All schools took part in 
some process evaluation activities. Findings were shared with the delivery team as they became 
available to provide formative feedback and facilitate ongoing development of the intervention. 


The cost of the intervention was estimated at £56,310, which includes up-front costs (£15,845) and 
running costs in the first year (£40,465). The cost per pupil was estimated at £27.13, based on 2,075 
pupils in total across the five participating schools in its first year.The cost per pupil was relatively low 
and is estimated to decrease in the programme’s subsequent years, with the estimate for the 
programme’s third year being £16.80 per pupil per year. 


Key findings 


The table below summarises the key findings. 


Question tate tare} Comment 


Is there evidence of No There was no evidence that teachers’ attitudes 
promise? towards research, or their use of research evidence in 
teaching practice, changed during the intervention. 


Was the approach The programme ran largely as intended in terms of 


feasible? activities and outputs, but teacher engagement and 
attendance at events was lower than expected. There 
were mixed views on the usefulness of some elements 
of the programme. 


Is the approach ready to The programme needs to be developed further before 
be evaluated in a trial? being evaluated in a full trial. 
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Introduction 


This study aimed to evaluate the Ashford Teaching Alliance Research Champion* programme, a 
school-based intervention delivered in the Ashford, Kent area. The project was funded by the 
Education Endowment Foundation as part of the Research Use in Schools round of projects* and was 
delivered during the 2014/2015 academic year. 


Intervention 


The programme was an intervention piloted in five schools—four secondary and one primary. Three 
schools were in Ashford, a town in the county of Kent in the South East of England, and two in 
neighbouring towns with rural catchment areas. A sixth school (a primary), originally recruited to be 
part of the programme, left during the early stages after a change of leadership. Participating schools 
were either already part of the Ashford Teaching Alliance when the programme began or joined at the 
point of recruitment. 


The delivery of the programme was led by a teacher (the ‘Research Champion’) based at one of the 
five participating schools. The Research Champion had run a variant of the programme in the 
2013/2014 academic year: this laid the foundation for the project. 


Each participating school identified a research lead to take the programme forward for them. During 
the pilot year (2014/2015) the Research Champion worked with research leads and other teachers in 
participating schools—including senior leadership team members—to promote engagement with 
research evidence. 


The programme had four key components: 
1. School audits, to identify individual school’s needs and interests in terms of educational 


research, conducted by the Research Champion. 


2. A-series of research symposia (three over the course of the pilot year). The symposia were 
envisaged as staff development days where seminars from educational researchers were 
combined with workshops. The aims of each symposium were: 


e to provide access to insights and ideas from recent educational research; and 


e to create a forum in which to discuss the challenges of making improvements based on 
these insights. 


3. Termly Research and Development ‘twilight forums’ to provide a context to explore how to use 
research both in the classrooms and at the senior leadership level. The aims of each session 
were: 


e to provide a forum for discussing the challenges of evidence-based teaching; 
e to provide new ideas using examples drawn from recent educational research; and 
e to reflect on the themes covered in the research symposia. 


4. Bespoke research brokerage for each participating school (one session per school each half 
term). Support aimed to address the particular needs and requirements of each school, but 
was anticipated to include services such as the design and running of staff workshops, help 


* The programme was referred to as the ATA Research Based Education Project in some early project materials. 


: https://educationendowmentfoundation.org.uk/projects/projects-a-z/research-use-in-schools/ 
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with the practical implementation of new practices, and assistance to make better use of 
internally produced evidence. 


In addition to symposia, twilight forums, and bespoke research brokering sessions, the programme as 
originally set out was anticipated to include regular updates on the progress of the project to schools 
via a twice-termly printed newsletter. 


The intervention aimed to help staff at participating schools to: 
e understand the key findings of specific educational research; 


e apply educational research findings in the classroom and at a strategic development level; 
and 


e establish a stronger culture of evidence-based enquiry and practice. 


These three aspects were expected to help improve the quality of the teaching and learning in each 
school and, in the longer term, to positively impact pupil outcomes. 


Background evidence 


The gap between research evidence on one hand and policy-making and professional practice on the 
other has increasingly become the focus of attention in the UK as well as across the world (Cooper et 
al., 2009; Nutley et al., 2007). A number of studies have sought to define (research) knowledge 
mobilisation and propose ways for improving its effectiveness in education and overcoming the 
existing barriers (Cooper et al., 2009; Levin, 2011). 


The barriers to research engagement include skill issues (Such as an inability to interpret research 
findings), resource issues (Such as lack of time or access to academic publications), and insufficient 
rewards in the system (Bransford et al., 2009; Nutley et al., 2007; Sharples, 2013; Hemsley-Brown 
and Sharp, 2003). It has also been noted that research evidence needs to be transformed before it 
can be used in teaching practice: this involves more than simply summarising it and requires effective 
collaboration between teachers and researchers and/or ‘mediation’ (Nelson and O’Beirne, 2014). 


This programme aimed to communicate research evidence to teachers in a form that could be more 
easily used in their practice (through, for example, presentations at the training events), as well as 
building capacity among the teachers to try new approaches with the support from the Research 
Champion. 


This project is one of the EEF’s Research Use in Schools funded projects. These projects were 
funded to explore knowledge mobilisation in the teaching profession, and consider how research 
evidence is integrated into teachers’ practices and school processes. The projects funded under this 
round were designed to explore three key questions: 


e How can research organisations and others effectively communicate their findings and 
engage with schools? 

e How can schools overcome the barriers to using research well? 

e How can brokers and mediators help schools find and use evidence-based approaches? 


In addition to this programme, four other projects were funded as part of the Research Use in Schools 
funding stream:° 


’More detail on these projects can be found at https://educationendowmentfoundation.org.uk/projects/projects-a- 
z/research-use-in-schools/ 
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1. Research into Practice — Evidence-informed Continuing Professional Development in 
Rochdale: a pilot project in ten primary schools in Rochdale area, testing whether the 
evidence-informed CPD programme in Rochdale is a feasible model. 


2. Research Learning Communities: an efficacy trial of a project developed by the Institute of 
Education to examine whether ‘evidence champions’ are effective at promoting research use 
in schools when supported by a research community of peers from local schools and an 
academic facilitator. 


3. The Literacy Octopus — Communicating and Engaging with Research: a large multi-arm 
randomised controlled trial investigating a range of different methods of communicating 
research to schools and engaging them in research evidence. 


4. The RISE Project — Evidence-informed school improvement: an efficacy trial of a project led 
by Huntington School that aims to test whether a research-based school improvement model 
makes a significant difference to classroom practice and student outcomes. 


This report presents findings of the first evaluation of the Research Champion programme. This is a 
pilot study, and in the conclusions of the report we comment on whether this programme is ready to be 
evaluated in a trial. 


Evaluation objectives 


The evaluation of the ATA Research Champion programme pilot, like other projects in the EEF’s 
‘research use’ round, had the following key objective: 


e to explore whether, and to what extent, research communication and engagement strategies 
had the potential to improve teachers’ use of, and attitudes towards, academic research to 
support pupils’ progress. 

Other evaluations in this funding round looked at the impact of different research communication and 
engagement strategies on pupil attainment. Although analysis of KS2 and KS3 data was part of the 
original protocol for this evaluation, both the evaluator and the EEF decided not to carry out this 
analysis. This was for two main reasons. First, given the slow progress of the intervention in the 
beginning (the first symposium took place in December 2014) it was felt that there would not be 
sufficient time for pupils to experience the benefits of the intervention by the time outcome data was 
collected in the summer term, especially as improvements to pupil attainment were a longer-term 
impact in the intervention’s logic model (See Figure 2). Second, the intervention activities did not 
necessarily target the year groups for which KS data would be available. 


Project team 


The intervention was delivered by the Research Champion, a teacher based at one of the five 
participating schools in and around Ashford. 


The independent evaluation team was led by Dr Svetlana Speight at NatCen Social Research who 
was supported by Dr Julia Griggs, Dr Javiera Cartagena Farias, Alexandra Fry and Dr Triin Edovald. 
The evaluation team was responsible for the design and delivery of the evaluation, including a logic 
model workshop, baseline and outcomes surveys of teachers, interviews with staff, observations of 
training events, and analysis and reporting. Questionnaires for the baseline and outcomes surveys of 
teachers were developed by the National Foundation for Educational Research (NFER) (Poet et al., 
unpublished) and used across all projects in the ‘research use’ round. 


Ethics 


The project underwent an ethical review by NatCen’s Research Ethics Committee and was approved 
in September 2014, prior to data collection. 


The headteacher at each school signed a Memorandum of Understanding that included consent to 
provide the Research Champion with the names and email addresses (and phone numbers where 
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appropriate) of all teachers at their school for passing on to the evaluation team (see Appendix B). In 
addition, written information about the study was sent to headteachers at the participating schools by 
post prior to the baseline survey taking place (see Appendix C), and information for teachers about the 
surveys was sent to them by email (Separately for the baseline and outcomes surveys; see Appendix 
C). Information about process evaluation activities was provided to participants by researchers 
undertaking interviews and observations in advance via email and again face-to-face or by telephone 
at the time of data collection, and consent was sought prior to interviews and observations taking 
place. Information about the study was also available on the study webpage on the NatCen website, a 
link to which was included in communications with research participants. 
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Methods 


Recruitment 


The five schools participating in the programme were recruited by the Research Champion. This 
included the Research Champion's own school and two other schools that were existing members of 
the Ashford Teaching Alliance and were previously involved in the programme. Additionally, two 
further local schools were recruited to the project (and also to the ATA) in order to increase the 
diversity of those included in the pilot. In particular, the two additional schools were recruited to ensure 
schools taking part in the programme had different pupil populations, for example, proportions of 
pupils eligible for free schools meals (FSM). The Research Champion used existing contacts to secure 
involvement of the final two schools, and ensured that consent was secured from senior leaders at 
each of them. 


The Research Champion worked primarily with research leads within each of the schools. There was 
no fixed process for recruiting research leads, with some staff self-selecting and others being put 
forward by senior leaders. 


Research leads were identified during the early stages of the programme, and contact information was 
supplied to NatCen by the Research Champion. The first round of interviews (with research leads and 
the Research Champion) took place in October and November 2014. The second round of interviews 
took place with other teachers at participating schools: these were also involved in the programme but 
were not the research leads. Teachers were selected and recruited to take part in additional staff 
interviews by the Research Champion. All staff selected for interview had attended at least one of the 
events. The aim of these additional interviews was to provide a more complete view of the programme 
from the teachers’ perspective. The research leads from each school were invited to take part in a 
follow-up interview in the summer of 2015. Two follow-up interviews were also conducted with the 
Research Champion. 


While the overall response to interview invitations was good, there were difficulties scheduling 
interview appointments given restrictions on teachers’ time. It was not possible to interview the 
research lead in one of the schools at either round of interviews. 


For the baseline survey of teachers, the evaluation team asked the Research Champion for a list of all 
teachers (including headteachers and those in management positions) at the five participating 
schools, together with their contact details. All teachers were then invited to take part in the baseline 
survey. Those who completed the baseline survey were eligible for, and invited to take part in, the 
outcomes survey at the end of the academic year. The repeated survey allowed the evaluation team 
to explore changes in attitudes and behaviours over the course of the programme. 


No data was collected from or about pupils at the participating schools. 
Data collection 


The evaluation of the programme was a mixed methods (qualitative and quantitative) study. Findings 
from the evaluation were shared with the delivery team as they became available in order to provide 
formative feedback and facilitate ongoing development of the intervention. 


As part of the evaluation, several methods were used, including: 


e the development of a logic model; 
e _pre- and post-intervention surveys; and 


e in-depth interviews and observations. 
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The original design for the evaluation also included analysis of pupil attainment data from the National 
Pupil Database because one of EEF’s key objectives is to improve pupil attainment. The intention was 
to measure pupil progress using Key Stage 2 (for primary schools) and Key Stage 3 (for secondary 
schools) attainment data, and to compare pupils in participating schools with a matched comparison 
group of pupils in schools not taking part. However, it was decided in consultation with the EEF that 
assessing pupil progress would not be an appropriate measure of the programme’s success, or 
potential, at this stage in its development. In particular, it was felt that pupil progress was a longer- 
term goal for this particular project, rather than a short or medium term one (as set out in the logic 
model). Also, the process evaluation revealed that the pupils who were directly affected by the 
intervention belonged to different year groups, including pupil groups not captured by Key Stage 
attainment data. Moreover, since the programme got off to a slow start it was felt that there would be 
insufficient time for pupils to experience the benefits by the time Key Stage data was collected in the 
summer term. These factors meant that any change in pupil attainment identified, positive or negative, 
could not be linked in any way to the programme. Consequently, analysis of pupil attainment data was 
not undertaken. 


Logic model 


A logic model workshop was held with the Research Champion in September 2014 to identify the 
resources, activities, outputs, and intended outcomes of the programme. The logic model was then 
drafted by NatCen researchers and finalised in consultation with the Research Champion (see Figure 
2). A copy of the logic model was provided to the Research Champion for sharing with participants at 
the training events. 


Surveys 


A key objective of the evaluation was to explore whether the programme demonstrated the potential to 
improve teachers’ use of, and attitudes towards, academic research. This was assessed using 
baseline and outcomes surveys developed by the NFER for the EEF (See Appendix A) specifically for 
use in all of the ‘research use’ round evaluations (Poet et al., unpublished). 


The surveys were administered online by NatCen Social Research with teachers in all intervention 
schools at two time points: during the start-up period (baseline) and in the intervention’s final school 
term (outcomes). All teachers at the participating schools were invited to take part in the baseline 
survey. 


A letter was initially sent to the headteacher of each participating school in September 2014 to inform 
them of the upcoming survey and to ask for their support in encouraging teachers to complete it. 
Individual teachers were then sent an email that included a weblink to the survey; this was followed by 
a maximum of three email reminders. The baseline survey took place in September and October 2014 
and the outcomes survey in June and July 2015. All teachers who had participated in the baseline 
survey were invited to take part in the web-based outcomes survey. This was followed by up to four 
reminder emails. The outcomes survey closed at the end of the summer term 2014/2015. 


Outcome measures 


NFER guidance (Poet et al., unpublished) recommends using six outcome criteria when analysing 
survey data collected in the EEF’s Research Use in Schools funded projects: 

1. positive disposition to academic research informing teaching practice; 

2. use of academic research to inform selection of teaching approaches; 

3. perception that academic research is not useful to teaching; 

4. perception that own school does not encourage use of academic research; 
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5. active engagement with online evidence platforms; and 
6. research knowledge. 


However, it was not possible to use outcome measure 6 as the questions were not asked at baseline. 
The other five outcome measures were constructed using relevant survey variables and following the 
NFER guidance (see Appendix A for details of the measures). The Cronbach’s alpha score, and 
therefore the reliability of the measure, varied considerably between the five scales. The results are 
displayed in Table 1. 


Table 1: Cronbach’s alpha scores for the five composite measures 


Outcome measure Cronbach’s alpha Cronbach’s alpha | Reliability of the 
in baseline survey in outcomes measure 
survey 


Measure 1: Positive disposition to 
academic research informing 0.86 ; high 
teaching practice 
Measure 2: Use of academic low at baseline, 
research to inform selection of 0.37 . moderate at 
teaching approaches outcomes 
7 


Measure 3: Perception that 
academic research is not useful to 0.60 0.72 moderate 
teaching 


Measure 4: Perception that own moderate at 
school does not encourage use of 0.71 0.37 baseline, low at 
academic research outcomes 
Measure 5: Active engagement with 
: ; site 0.61 moderate 
online evidence platforms 


The Cronbach's alpha scores for measure 4 were considerably lower than for the others—scores so 
low as to suggest the internal consistency was very poor and the measure was not reliable. For this 
reason measure 4 has not been used in this report. Instead, we analysed the two separate questions it 
was intended to be based on. 


Approach to survey analysis 


Analysis focused on change over time between the two surveys. There was no comparison group so it 
was not possible to compare the changes to a ‘business as usual’ scenario. The main outcomes of the 
programme were estimated using descriptive analysis, frequencies and cross-tabulations, and paired 
sample t-tests.* Cronbach's alpha was used to assess the internal consistency of the outcome 
measures, and McNemar’s test to explore whether differences between responses at baseline and 
follow-up were statistically significant for individual questions. 


Differences between estimates which were significant at p < 0.05 are referred to throughout the report 
as being ‘statistically significant’, and those which fall below the 5% threshold as non-significant. 
However, it is important to note that as the total sample size available for analysis of outcomes was 
fairly small (n = 106), it limited our ability to detect significant results. All analysis was conducted in 
IBM SPSS Statistics for Windows, Version 21.0. 


*“ Measures of effect size were not considered appropriate given the absence of treatment and control groups. 
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Process evaluation 


The process evaluation was designed to explore views on delivery, implementation, and perceived 
impacts of the programme. A mixed-methods approach was used to meet these objectives. This 
included observations of two research symposia and one twilight forum, interviews with research leads 
and other teachers at the participating schools, and survey items specific to the Research Champion 
programme for participating teachers (see Appendix E Block F). 


Depth interviews with research leads and teachers in the five schools, as well as with the Research 
Champion, (most of which were conducted by phone) took place at three time points: 


1. November—December 2014: interviews with the Research Champion and four of the five 
school research leads. These interviews focused on the objectives of the intervention and 
early implementation processes. 


2. March 2015: follow-up interview with the Research Champion. 


3. May-July 2015: follow-up interviews with the Research Champion and school research leads 
(four of five), as well as interviews with other teachers involved in the programme at each of 
the five schools (11 teachers in total). 


The inclusion of interviews with the Research Champion in the design meant that information was 
gathered from the perspectives of both the programme designer/deliverer and the participating school. 


The content of each interview was based on a topic guide to ensure systematic coverage of key 
issues. It was intended to be flexible and interactive, allowing individual respondents to discuss issues 
of particular relevance to them. The interviews covered: 


1. the profile of the school and background information about the teacher, including their role 
within the programme; 


2. how the programme had been implemented in the school; 
views on the programme and any recommendations regarding how it could be improved; 


4. assessment of whether the programme had had an impact on the interviewee’s teaching 
practice, and on pupils’ behaviour, attitudes, and/or attainment; and 


5. barriers to the implementation of the programme and to sustaining it within the school beyond 
the pilot. 


Interviews were digitally recorded and full interview notes taken. All participants were told that 
everything discussed in the interview would remain confidential and would be treated in accordance 
with the Data Protection Act, and that their views and opinions would only be reported anonymously. 
However, it was made clear that details about how the programme had been implemented in each 
school might be included in the report. 


In addition to the qualitative interviews with teachers, the outcomes survey included a series of 
process questions that asked those teachers directly engaged with the project to provide feedback on 
the programme and its different components. Findings from this part of the survey are discussed in the 
process evaluation section. 


Figure 1: Timeline of the evaluation 
Date 


Summer 2014 Schools recruited to the intervention 


September-October 2014 | Programme delivery began 
Logic model workshop 
Baseline survey of teachers 
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November-—December First phase of process evaluation interviews with research leads and 
2015 the Research Champion 
Observation of first symposium 


March 2015 Observation of twilight event 
second interview with the Research Champion (of three) 


May-July 2015 Final stage of process evaluation interviews with research leads and 


the Research Champion 
process evaluation interviews with additional staff at participating 
schools 


June-July 2015 Outcomes survey of teachers 
observation of third symposium 

August 2015-January Analysis and reporting 

2016 


Costs 


Information on the costs of the intervention was collected from the Research Champion. Our estimate 
of the total cost includes the up-front costs (Such as a programme launch meeting), the costs of 
providing the training and support, and the costs of teachers attending the intervention events. The 
costs of developing the intervention are not included. 


This intervention was implemented at the whole-school level and intended to benefit pupils across all 
year groups, however the implementation was gradual with some pupils receiving the direct effect of 
the intervention only at the end of the academic year. Therefore, to calculate the total cost per pupil in 
the intervention’s first year we considered only those pupils who were involved in the programme from 
the beginning of the academic year (n = 2,075). When estimating costs per pupil in the intervention’s 
second and third years we assumed that all pupils at each school would be affected by the 
intervention from the beginning of the academic year, and thus used the total number of pupils across 
the five participating schools (n = 4,851). 
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Findings 
Participants 


Five schools (four secondary and one primary) were recruited to participate in the project. Three of the 
schools were based in Ashford, a town in the county of Kent in the South East of England, and two in 
neighbouring towns with rural catchment areas. A sixth school (a primary), originally recruited to be 
part of the programme, left during the early stages after a change of leadership. The remaining 
schools stayed involved during the pilot year. 


The four secondary schools had an average of 1,162 pupils,” higher than the national average for 
secondary schools (957 pupils in 2014): three were larger than the national average, and one smaller. 
The one primary school was slightly smaller than the national average for primaries (263 in 2014). 


The proportion of pupils eligible for free school meals (FSM) among the secondary schools was 26.1% 
on average, slightly lower than the national average of 28.5%." In contrast, the one participating 
primary school had a considerably higher proportion of FSM pupils than the national average for 
primary schools (26.6% in 2014). Overall, there was a high level of variation in FSM eligibility between 
schools. 


The schools were also mixed in terms of their Ofsted ratings: one of the schools was rated as 
‘outstanding’, another as ‘requires improvement’, and the remaining three as ‘good’. 


Key characteristics for each of the schools are displayed in Table 2. 


Table 2: School characteristics 


rsTo4 Toe) Type Urban! rural FSM 


eehecia Primary Uicban Slightly smaller Higher than 
than average average 


Slightly larger than | Lower than 


School 2 Secondary average average 


average average 
average average 
average 


As the programme aimed to achieve change at the whole-school level rather than only for teachers 
and pupils directly involved in the programme, all teachers at the participating schools were invited to 
take part in the teacher surveys at baseline and at the end of the academic year. When reporting on 
findings of these surveys, we present estimates both for all teachers and for different subgroups 
depending on their level of involvement. 


The overall response rate for the survey was 63% at baseline (190 of an eligible 304 respondents) and 
56% for the outcomes survey (106 of 190 eligible respondents). Those responding to the survey were 
predominantly either classroom teachers (41%) or middle leaders (40%). A further 15% held senior 
leadership positions or were school heads, while 5% recorded their job as ‘other role’. In the outcomes 
survey, there was a good distribution of respondents with regard to teaching experience, ranging from 


° Figures have been drawn from each school’s Ofsted reports, which, in turn, used the 2014 January census. 


° For national statistics, see https://www.gov.uk/government/organisations/ofsted 
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NQTs (5%) to those with more than 20 years’ experience (19%). More than half of the teachers 
surveyed had some involvement in the programme, be it first-hand (17%) or via colleagues (31%). 


Logic model 


A logic model workshop was held with the Research Champion at the beginning of the evaluation to 
identify the resources, activities, outputs, and intended outcomes of the programme. The logic model 
was then drafted by NatCen researchers and finalised in consultation with the Research Champion 
(see Figure 2). A copy was provided to the Research Champion to be shared with participants. 


The logic model informed the development of topic guides for the process evaluation, however it was 
not used for the construction of data collection instruments for the quantitative element of the 
evaluation (the survey). This is because the survey questionnaires were developed by the NFER to be 
the same for all projects in the ‘research use’ round (see Appendix D). As a result, the teacher- and 
school-level outcomes measured in the baseline and outcomes surveys do not precisely match the 
ATA Research Champion logic model. 


The delivery of the programme, and any deviation from the logic model, is discussed in full in the 
process evaluation section of this report. To summarise, findings indicate that the activities set out in 
the logic model were largely delivered as intended. Attendance at the events, however, was not as 
high as expected (three teachers from each of the participating schools were expected to attend all 
three symposia, and two from each school to attend the six twilight forums). While the research 
symposia were well attended overall, few of the teachers from participating schools were able to 
attend more than one event.’ Attendance at the twilight forums was also slightly lower than 
anticipated,8 with some schools not being represented at all at some of these. 


There were also more mixed results in terms of engagement with the research brokerage service: 
some schools made good use of the provision while others struggled to find sufficient time for the 
meetings. The structure of this service also changed post-implementation. While it was originally 
anticipated that schools would schedule a day’s consultation with the Research Champion each half 
term, in practice shorter, more regular meetings were found to be more practical (in terms of staff 
availability) and to better support a coaching/mentoring approach. 


Additionally, the twice-termly newsletter (as set out in the logic model) was only delivered at the end of 
the first half term. The very disparate activities being carried out at each of the schools made it difficult 
to create a regular newsletter that would be relevant for all participating schools. It was substituted 
with updates linked to the symposia and twilight forums, shared articles, and research summaries. The 
activities and outcomes outlined in the logic model (Figure 2) are discussed in further detail in the 
process evaluation section below. 


’ Attendance figures for symposium or twilight events were not collected in a systematic way. 


® An observation was carried out at the March 2015 twilight forum. A total of nine delegates attended from four of 
the participating schools. 
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Figure 2: The ATA Research Champion project logic model (based on six schools taking part) 


Your planned work 
Resources Activities 


Endorsement from Design and 
EEF implement 
research needs 
audits in schools 


Endorsement from 
participating 
schools Design and 
implement 
research symposia 


Nominated 
research leads’ in 


each school 
Design and 


implement termly 
Research and 
Development 


First year’s fundin: 
¥ 9 twilight forums 


for the Research 
Champion Project 


Design and 
implement 
bespoke research 
brokerage for each 
individual schoo! 


Design and 
implement a 
dissemination 
strategy for regular 
updates 
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Short-term (up to lyr) 


Increased interest in 
research findings 


Improved understanding of 
research evidence 


Increased knowledge of 
where to find evidence 


Improved schooHevel 

understanding of using 

evidence in improving 
practice 


Your intended results 
Outcomes 


Medium-term (1-2 yrs) 


Increased use of evidence 
in decision-making 


Improved teaching practice 


Improved classroom 
management 


Increased use of evidence 
in schooHevel decision 
te Ltlae| 


Increased school-level 
support for using evidence 
in educational practice 
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Long-term (2>yrs) 


WB OBB 0) Teacherischoo! outcomes 


la B Hae Pupil outcomes 
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Evidence to support theory of change 


Each section below looks first at the overall score for each outcome measure, including any change 
between the baseline and the outcomes surveys. It then considers differences between groups of 
teachers according to their level of involvement in the programme (based on the teachers’ own 
responses to the outcome survey rather than on any pre-allocation to one of the two groups). Finally, 
this section explores the individual questions that make up each composite indicator. 


All analysis has been conducted on data collected from teachers taking part in both the baseline and 
outcomes surveys. Where teachers took part in the baseline survey only, their data was not analysed. 


Positive disposition to academic research in informing teaching practice 


Results of the first composite measure showed very little change in teachers’ disposition to academic 
research in informing teaching practice, with a total mean value of 20.1 (SD = 4.0) at baseline and 
19.9 (SD = 4.0) at follow-up. 


Looking separately at groups of teachers according to their involvement in the project—(a) those 
involved directly or indirectly through colleagues,° and (b) those not involved, or who responded ‘not 
sure’ when asked about their participation—likewise showed very little difference (see Table | 


Table 3: Positive disposition to academic research in informing teaching practice by level of 
involvement 


Baseline (mean Outcome (mean Observations 
Yolo) «=)) Xero) <=) ()) 


First-hand involvement or 20.6 20.2 49 
colleague/s were involved and 
shared learning 


All teachers 20.1 19.9 


Note: ‘All teachers’ (N = 106) includes those who did not answer the survey questions about their level of involvement in the 
programme. 

Looking at each of the individual questions across the baseline and outcomes surveys revealed some 
interesting trends in terms of teachers’ attitudes towards research to inform teaching practice. For 
example, 72% of teachers reported finding it ‘easy’ or ‘very easy’ to understand ‘articles, reports, 
books or summaries based on academic research’ at baseline; this showed a notable—though not 
statistically significant—drop to 67% in the outcomes survey." 


9 Teachers with first-hand and indirect involvement in the programme have been grouped for this analysis as base sizes are 
very small (17 for those with direct involvement). 


10 When the same variables were included in an OLS regression model, with the baseline score as a control variable and 
outcome score as the dependent variable, there was no statistically significant difference by level of involvement in the 


programme. 


11 Frequencies exclude the missing cases and the respondents who said that they ‘did not use this source’. All have been 
included in the composite outcome variable. 
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Likewise, the proportion of teachers who agreed or strongly agreed that that they knew where to find 
relevant research fell from 54% to 52%, as did the proportion of teachers who reported feeling 
confident about analysing information from research (63% at baseline falling to 54% at follow-up). The 
proportion of teachers who reported that they used information from research to implement new 
approaches in the classroom also fell slightly from 60% to 59%. However, none of the changes were 
statistically significant. 


In contrast, the proportion of teachers who were able to relate research to their own context showed a 
statistically non-significant increase between the baseline and outcomes surveys (from 61% to 64%). 


Proportions were constant across the baseline and outcomes survey for those reporting that research 
‘plays an important role in my/our teaching practice’ (57%) (See Table 4). 


Table 4: Response to individual items included in question 8: ‘How much research is used at 
work?’ 


Baseline - Outcome -— Observations (N) 
strongly agree / strongly agree / 
agree (%) agree % 
Articles, reports, books or 
summaries based on academic 
research easy to understand 


Know where to find relevant 
research 


Feel confident about analysing 
information from research 


Use information from research to 
help implement new approaches 
in the classroom 


Able to relate research to own 
context 


Research plays an important role 
in teaching practice 


Use of academic research to inform selection of teaching approaches 


Results of the paired sample t-test showed that results remained stable between baseline and follow- 
up (at 4.1, SD = 1.2). Exploration of differences by level of involvement showed that while teachers 
who were involved in the project saw a small drop in their mean score, those with no involvement, or 
who were not sure, saw a slight rise in their outcome score (see Table 5 below). Neither result was 
statistically significant. *” 


When this association was explored using OLS regression (as with measure 1), no statistically significant 
difference was found in the outcome score by level of involvement in the programme. 
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Table 5: Use of academic research to inform selection of teaching approaches 


Baseline — mean Outcome - Observations 
Tore] c=) mean score ‘(N)) 


First-hand involvement or 4.3 4.2 
colleague/s were involved and 
shared learning 


Note: ‘All teachers’ (N = 101) includes those who did not answer the survey questions about their level of involvement in the 
programme. 


Looking at the individual items that comprised the composite measure across the baseline and 
outcomes survey showed mixed results. While there was a large significant increase in the proportion 
of teachers who found ‘articles, reports, books or summaries based on academic research’ important 
in shaping their approach (a rise from 2% to 18%), differences on the other questions were not 
statistically significant. One of the items included in the measure, ‘online evidence platforms or 
databases important in informing choice of approach’, increased (non-significantly) from 2% to 6%. 


In contrast, the proportion of teachers who said that their decision to adopt a specific approach was 
‘strongly influenced’ by its backing by academic research fell from 36% to 26% (this change was not 
statistically significant). The proportion of teachers saying that they consulted academic articles, 
reports, books or summaries ‘a lot’ also showed a considerable drop, in this case from 18% to 9% (p = 
0.078). 


Perception that academic research is not useful to teaching 


Once again the mean score on this measure remained stable across the baseline and outcomes 
survey (mean 4.7, SD at baseline = 1.5, SD at outcomes = 1.3). 


Results were fairly consistent regardless of teachers’ involvement in the programme, with just a small 

(non-significant) drop in the overall score for teachers who were involved in the Research Champion 
: 13 

project. 


Table 6: Perception that academic research is not useful to teaching by level of involvement 


Baseline -mean Outcome-mean Observations (N) 


First-hand involvement or 
colleague/s were involved and 


shared learning 


Not involved, or not sure 


All teachers 


Note: ‘All teachers’ (N = 106) includes those who did not answer the survey questions about their level of involvement in the 
programme. 


*8 As with measures 1 and 2, the association was tested using OLS regression analysis. Results showed no 
statistically significant relationship between the outcome measures and level of involvement in the programme. 
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Each of the two component questions showed a change between the baseline and outcomes survey, 
but neither of the results reached conventional levels of statistical significance. The proportion of 
teachers who ‘agreed’ or ‘strongly agreed’ that they ‘do not believe using information from research 
will improve pupil outcomes’ fell from 7% to 2%. The change was smaller in the second item— 
‘research conducted elsewhere is of limited value to school’—where the proportion of teachers who 
‘agreed’ or ‘strongly agreed’ dropped from 10% to 9%. 


Perception that own school does not encourage use of academic research 


Composite measure 4 ‘perception that own school does not encourage use of academic research’ has 
been excluded from this report on the basis of its poor internal consistency (as discussed above). 
Instead, the evaluation included analysis of the individual questions brought together under this 
measure in the NFER guidance. 


Comparing baseline and outcome responses to each of the component questions showed a change 
towards a more positive perception of the use of evidence in the teachers’ own school on both 
measures. For example, the proportion of teachers who ‘agreed’ or ‘strongly agreed’ that ‘school 
leaders do not encourage use of research to improve practice’ showed a decline between the baseline 
and outcomes survey, from 16% to 9%. The proportion of teachers who ‘agreed’ or ‘strongly agreed’ 
that ‘other staff in my school rarely use information from research to inform their teaching practice’ 
also showed a drop (from 16% to 15%, see Table 7). However, neither change was statistically 
significant. 


Table 7: Perception that own school does not encourage use of academic research 


to use information from research to improve my 
practice 


Strongly agree 


WM WY ae=Xed avoXo) Mi (-r-Ce(-1a-4(efo)V(-1galele-me (em alelm-iileolele-le(-muil-) Baseline % Outcome % 


Strongly disagree 


Neither agree nor disagree 
3 
13 
32 
05 


oo 
— 


ther staff in my school rarely use information Outcome % 
from research to inform their teaching practice 
sromvaoe 
poe 
eiterpeenordingee 
poe 
seommvaseowe 


Note: percentages may not sum to 100 due to rounding. 
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Active engagement with online evidence platforms 


The average mean score for all teachers showed very little change over the course of the programme, 
from 4.3 (SD = 1.1) at baseline to 4.4 (SD = 1.1) at follow-up. This difference was not statistically 
significant. 


Exploration of the results by level of involvement in the programme also shows relatively little 
difference between the two groups of teachers. Once again the differences by level of involvement 
were not statistically significant. = 

Table 8: Active engagement with online evidence platforms by level of involvement 


Baseline -mean Outcome —- mean | Observations (N) 
rYox0) f=) Tore) (3 


First-hand involvement or 4.4 


colleague/s were involved and 
shared learning 


Not involved, or not sure 4.4 4.3 53 


| All teachers | 4.3 4.4 106 


Note: ‘All teachers’ (N = 106) includes those who did not answer the survey questions about their level of involvement in the 
programme. 


When considered separately, the two component questions showed mixed results (none of the 
differences reached conventional levels of statistical significance). While the number of teachers who 
consulted online evidence platforms and databases ‘a lot’ remained relatively stable between the two 
surveys, fewer teachers said that they did not consult them at all (a drop from 46% to 40%). As well as 
increased use, there was also a small rise in the proportion of teachers who reported finding it ‘quite 
easy’ or ‘very easy’ to understand online evidence platforms (an increase from 68% to 69%).*° 
However, the proportion of teachers who found online platforms ‘very easy’ showed a notable drop between the 
two surveys, from 16% to 10%. Likewise the proportion of teachers who did not find using online 
databases ‘easy at all’ rose from 4% to 8%. None of these changes over time were statistically 
significant. 


4 As with the other measures, the association between the outcome measure and level of involvement in the 
programme was explored using OLS regression analysis. Results showed no statistically significant association. 


*® Frequencies exclude missing cases and respondents who said that they ‘did not use this source’ (37 at baseline 
and 29 at follow-up). All have been included in the composite outcome variable. 
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Feasibility 


This section reports findings from the process evaluation of the implementation of the Ashford Teaching 

Alliance Research Champion programme, drawing on qualitative interviews with teachers and the 

Research Champion, observations of training sessions, and teachers’ answers to the outcomes 
16 

survey. 


The discussion of the feasibility of the programme is structured around its main components: 
1. audits of schools’ research interests; 
2. research symposia; 
3. twilight forums; and 
4 


research brokerage. 
Audits of schools’ research interests 


The intervention began with school audits intended to assess each school’s needs and interests in 
terms of research use, and to feed this information into a school-specific action plan. This component 
was included as one of the outputs in the logic model. In the interviews, teachers said that they found 
the early meetings and discussions with the Research Champion productive, and valued the 
opportunity to shape the implementation of the project in their own school. However, they were 
predominantly unaware of these discussions being a distinct phase of the project, or—as was more 
often the case—were aware of the assessment activity, but not under the audit label. 


The lack of a distinct ‘audit’ activity meant it was difficult for teachers to reflect on this component of 
the programme and therefore to assess the feasibility of this particular element. However, in general, 
teachers felt that the early meetings with the Research Champion met their needs, allowing them to 
reflect on the specific interests of their school and to consider the steps needed for them to implement 
learning in the classroom. 


In summary, this element of the programme was feasible, and appeared to broadly meet its objectives 
as set out in the logic model. However, further consideration is required as to whether the audit needs 
to be a distinct phase of the project, separate from subsequent research brokerage activity, and what 
specific activities distinguish the audit phase from research brokerage. Additionally, it would be worth 
considering how implementation of the audit could be made more consistent across schools, perhaps 
by developing a manual. 


Research symposia 


Three full-day research symposia were held over the course of the pilot, one per term. Teachers 
involved in the programme, and those belonging to non-intervention schools in the Ashford Teaching 
Alliance, were invited to attend (the latter subject to a £40 fee). The symposium events were directed 
by the Research Champion and included presentations and workshops led by expert speakers. 
Subjects were selected by the Research Champion, but were influenced by the interests of teachers 
directly involved in project. 


**© The number of teachers answering process evaluation questions in the outcomes survey was very low, as 
these were only asked of those with direct involvement in the programme. The number of responses ranged from 
8-14 depending on the question. 
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Each event was structured in the same way: introductory and closing sessions from the Research 
Champion, presentations in the morning session, and workshops in the afternoon. In this way events 
combined ‘taught’ sessions—which introduced research evidence on topics such as lesson study, 
feedback, and mindfulness—with more practical, applied workshops, where teachers were able to 
think about applying their learning in the classroom. Figure 3 provides an overview of the content of 
each symposium event. 


Figure 3: Research Champion project—symposium events 


Symposium Content 


Keynote seminar: ‘Learning from failure: how to 
improve over time’ 


Symposium 1 . Keynote seminar: ‘Formative assessment: can 
December 2014, 9:30-3:00 teachers really put it into practice?’ 
Afternoon workshops on_ strategies/models — for 


professional development and putting formative 
assessment and feedback into practice 


Keynote seminar: ‘Using Japanese lesson study to 
improve lesson design and student learning’ 


Symposium 2 . Keynote seminar: ‘What do we know about effective 
March 2015, 9:30-2:45 CPD and Professional Learning?’ 


Choice of afternoon workshops: (1) ‘How to evaluate 
and improve CPD provision across a school’, (2) ‘How 
to apply the principles of lesson study to day to day 
teaching’, and (3) ‘How to use lesson study in Maths 
teaching’ 


Keynote seminar: ‘Using evidence to narrow the gap: 

How can large-scale research help you to improve 

; things for your students? How can the principles of 

Symposium 3 research help you to evaluate new policies, 
July 2015, 9:30-2:45 programmes and initiatives planned for 2015/16? 


Keynote seminar: ‘Why is “mindfulness” trending? 
What does it offer? What does research say? How 
might it help your students and staff?’ 


Keynote seminar: ‘Mindfulness in practice’ 


Choice of afternoon workshops: (1) ‘Not everything 
works and why and that’s a good thing’ and (2) 
‘Mindfulness: a practical insight’ 


Analysis of the process questions included in the teacher survey found that the research symposia 
were viewed positively by participating teachers, with the majority of participants describing the 
sessions as ‘quite good’ or ‘very good’. Qualitative interviews with teachers indicated that most had 
been able to attend at least one of the three symposia. 
Positive aspects identified in feedback 
Teachers involved in the programme highlighted a number of positive aspects of these events: 

1. Expert speakers and opportunities to reflect on practice 


Teachers felt that a key barrier to the use of research evidence in the classroom was the time required 
to read and process the latest evidence. Providing targeted information about educational research at 
the symposia via ‘high quality’ speakers was felt to be an effective, efficient way for teachers learn 
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about evidence. Teachers found the topics covered thought-provoking, and reflected that the 
symposia prompted them to consider their own practice in the classroom, and how it could be 
developed. 


‘To have such high quality input; it’s better than any day up in London on CPD. Eons above that’ 
(research lead). 


2. Practical application of in-depth learning 


The combination of theory and practical application, through a ‘taught’ morning session and applied 
afternoon workshop, was seen as a particular strength of the symposia. Teachers appreciated the way 
symposia were tailored to provide practical ‘take away’ lessons, but which drew on detailed and robust 
research. Teachers highlighted the CPD workshop as a particularly good example of this: 


‘It was the combination of the high quality input, then the chance to go and reflect and think about how 
it relates to my own context and school. I'd really recommend them’ (classroom teacher). 


3. Opportunities to network and share ideas 


The symposia provided useful opportunities for participating teachers to network, build contacts, share 
experiences, and learn from one another. They valued the opportunity to meet and share practical 
experiences. Teachers appreciated that the ideas tabled came from other teachers rather than from 
those in authority. In one case, two participants who taught the same subject at different schools were 
able to build a rapport and arranged a school visit as a direct result of meeting at one of the symposia 
workshop sessions: 


‘We arranged that | would visit the other modern languages teacher in her school... It’s something to 
go and do as a consequence of the workshop’ (classroom teacher). 


4. Useful reference materials 


Teachers were given a pack of materials at each symposium to take away with them, which they 
found useful as a reference document to follow-up after the event. A small number of teachers made 
good use of this material, using it to search out references to evidence as directed by speakers. This 
meant that teachers were able to continue to engage with the symposia materials when they were 
back in the classroom. 


Negative aspects identified in feedback 


Alongside the positive feedback about the symposia, a minority of teachers also highlighted a number 
of areas where this component of the programme could be further developed: 


1. More applied sessions and clearer linking with applications in the classroom 


Although many teachers were happy with the balance between theory and practical application there 
were comments that workshops in particular could have been more applied and interactive, and also 
that at times they felt overly academic. Teachers talked about wanting an outcome, ‘product’, or 
concrete plan at the end of the day that they could take back to school and apply. One teacher 
suggested that participants could have brought along their school’s CPD plan to the workshop. They 
could then have discussed the plans and developed them while there. Although this would not result in 
a polished product, it would be more productive and efficient. It was thought that while teachers would 
come away from the events meaning to work through their notes and implement the things they had 
learned, that day-to-day school life would make this difficult. 


2. Making the Research Champion's role at the events more clearly defined 


It was felt that although the Research Champion coordinated the events very effectively, there was 
some confusion as to whether he should be viewed as a fellow teacher or a leader. One teacher 
suggested that this was something that could be made clear at the start of each symposium. 


3. Clearly define the goals of the symposia in advance 


Although the content of the symposia was clearly presented in the materials supplied to teachers, it 
was felt that it would be helpful to have a clear sense of the overarching goal of each symposium, and 
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the events as a whole. In other words, to have a stronger focus on what teachers would take away 
from the symposia at the outset of the programme. 


To sum up, the research symposia element of the intervention was delivered largely as intended, was 
feasible, and was positively received by the participants. 


Twilight forums 


In addition to the research symposia, research leads and colleagues involved in the programme from 
each of the participating schools were invited to attend twice-termly ‘twilight forums’. These were held 
at the end of the school day at one of the participating schools, and led by the hosting school. They 
were intended to offer teachers an opportunity to share experiences and reflect on a specific topic, 
these included feedback and assessment, questioning, and lesson planning. 


Teachers found these events productive, offering opportunities to discuss learning in practice in a 
supportive environment, to reflect on research areas covered at the symposia, and share ideas and 
good practice. The smaller groups of those more directly involved in the project brought a sense of 
unity to the events, and gave teachers the opportunity to talk in more detail about their own work. 
Participants particularly valued hearing first-hand about the experiences of schools and teachers in 
implementing lessons from research, although they recognised that this was not always directly 
transferable to their own educational context. This particularly applied to the one primary school taking 
part in the pilot that felt that lessons were more relevant for secondary schools. Teachers accepted, 
however, that some topics would be more relevant to their own schools (and subjects) than others, 
and sought more generic learning points to take back with them: 


‘What’s interesting is when you get a group of teachers from a range of different subjects, we 
all have very different needs and focuses, so it is interesting hearing what everyone’s got to 
say, and obviously some bits you can discard because they don't apply and there are other 
things that do’ (classroom teacher). 


The positive comments about the twilight forums were reflected in the outcomes survey. Results 
reveal a high level of satisfaction, with the majority describing them as ‘very good’ or ‘quite good’. 


Despite generally positive comments, some teachers also offered constructive feedback, including 
suggestions of areas for development. 


It was noted that attendance at the twilight forums was not as good as hoped, and some teachers 
suggested that the format was problematic. (Although one teacher felt that low numbers were an 
advantage.) This was felt to be connected to the timing of sessions, specifically, that teachers were 
required to travel to another school after a day’s teaching. Participating schools sometimes held staff 
meetings at the same time as the twilight forums and failed to release staff. 


‘Due to perhaps the logistics of having been a day in school and then driving to a different 
location and then having one hour... With the symposium, you went there in the morning, you 
were fresh, you could really engage and think about the ideas and what you were being 
presented with and then get your teeth into it, whereas the twilights just felt like a bolt-on’ 
(research lead). 


Additionally, as with the symposia, it was felt the twilight forums could sometimes be overly academic 
in focus, and that the focus or goal of the forums was not always clear at the outset. 


Feedback from the Research Champion suggested that the twilight forums had been less effective 
than envisaged, primarily due to issues with attendance and with their ability to cover new ground in 
terms of topic. The scheduling of twilight forums had proved problematic for teachers despite early 
discussions about timetabling and advanced notice regarding dates. The format had been tweaked 
going forward to include just the research leads at each of the schools. It was felt that the smaller, 
more focused events would improve effectiveness. 
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In sum, the evidence as to whether twilight forums were implemented as originally conceived, and 
whether they were feasible, was mixed. More thinking is required about the purpose of this element of 
the intervention and about how it could be implemented in the most effective way, in particular, around 
the format and scheduling of the twilight events and expectations in terms of attendance. 


Research brokerage 


In addition to the events, the Research Champion programme included a bespoke research brokerage 
service available to all participating schools. The brokerage sessions were intended to be an 
opportunity for the Research Champion to share evidence on topics of interest to individual schools 
and to support the practical implementation of research evidence in the classroom. The particular 
research topics covered in the research brokerage sessions were identified by the schools 
themselves. Originally envisaged as a one-day visit per half term, the model was quickly adapted to a 
larger number of shorter visits. The model also included informal ad hoc advice by email or telephone. 


Research leads generally made good use of the research brokerage service, meeting with the 
Research Champion several times over the course of the pilot year. Brokerage also included sessions 
with small groups of staff in all participating schools, and with all teaching staff in two schools. 
However, the number and frequency of meetings varied from school to school, as did teacher 
engagement. Teachers also utilised the ad hoc support (although again to varying degrees) and 
appreciated having a source of advice and expertise on hand. 


Interviews with teachers showed that they valued: 


1. having an informed outsider, questioning the approach and theory of change in a supportive 
and constructive way; 


2. the emphasis on reflective practice; 


the momentum the visits added to the programme, keeping it prominent in teachers’ minds 
when the day-to-day practice of teaching threatened to take over; and 


4. having a source of support and advice available and helpfully responsive when called upon. 


As with the research symposia and twilight forums, the teacher survey found a reasonably high level 
of satisfaction with the research brokerage service, with the majority of teachers describing it as ‘very 
good’ or ‘quite good’. However, a substantial minority found the brokerage service ‘average’, or ‘poor’. 
The survey also showed a more mixed response from participating teachers to ad-hoc support. 


The original plan for this component was to conduct one full-day visit per half term, however this was 
revised as the programme was implemented in that research leads and other staff were now visited on 
a more ‘as needed’ basis. These visits were shorter and more frequent. This approach was felt to be 
more appropriate as it allowed flexibility for staff and could more easily be accommodated with other 
teaching responsibilities. 


Regular meetings with the Research Champion were seen as important as a catalyst to moving the 
project forward. In this respect, more frequent visits increased the momentum of the programme. 
However, some teachers struggled with scheduling and stressed the importance of flexibility, for 
example, to be able to step back from the project during pressure points, such as Ofsted visits. 


To sum up, evidence relating to the research brokerage and ad hoc support components was mixed. 
While the service was generally well used and well received, there was variation in how, and how 
often, it was used across participating schools. Changes were made to the way this component was 
implemented, deviating from the original plan as set out in the logic model. However, these changes 
were made in response to the needs of participants and benefitted the programme. In the future, 
further thought could usefully be given to the best structure and format for the brokerage service, as 
well as to the optimum number and regularity of meetings. 
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Implementation in schools 


Each participating school identified a research lead to take the programme forward for them, and the 
Research Champion encouraged them to identify an area of research in practice that had meaning for 
their schools. As the schools varied considerably in terms of their pupil intake, and in more significant 
ways (one of the schools was a primary school), the approaches they took also varied. This was true 
both in terms of the topic/s explored and the extent to which new approaches were implemented. 


Indeed there was considerable variation among participating schools, ranging from one school which 
had regular meetings with the Research Champion but had yet to make changes to teaching practice, 
to another which had made multiple inroads into establishing and responding to research evidence. 
For example, one school had explored the literature looking at questioning, had conducted its own 
lesson observations and student voice sessions, and, in response to its findings, had introduced a new 
exercise book that allowed pupils to show rough workings alongside neatly presented final work 
(overcoming children’s anxiety about spoiling their exercise books with mistakes). This particular 
school differed from the wider group in that it had a teacher in the research lead role who had 
dedicated time to devote to the programme. The research lead was also working closely with a small 
team of staff and regularly shared planning time. 


Teachers were generally positive about implementing learning from the programme in the classroom, 
although the extent to which this happened varied considerably from school to school. Teachers 
flagged a number of facilitators to successful implementation during the interviews, these included: 


1. Context-specific implementation 


Research leads appreciated being able to select an area of research most appropriate for their own 
schools’ needs, and having the support of the Research Champion in this process. Such flexibility was 
viewed as integral to the success of the programme as it allowed schools to adapt it to their own 
specific context and pupil population. This was felt to be particularly important given the diversity of the 
schools. Additionally, the ability to focus on a topic pertinent to a particular school was felt to increase 
commitment to the programme and engagement with research evidence. 


‘Both the challenge and also the area where we can maybe have the most impact, is trying to 
link together the ideas that are available with specific local issues’ (Research Champion). 


‘This is the best form of professional development I’ve been on... the power of this is that you 
can make the process relevant for yourself’ (research lead). 


2. Implementation support from the Research Champion and others 


Research leads and other teachers felt that the combination of events and visits from the Research 
Champion worked well to facilitate classroom implementation. They were positive about the 
opportunities the different events gave them to share practice with other teachers and schools, 
particularly to discuss the successes and learning points from their own practical experience. This was 
true both of the wider events, such as the twilight forums, and the in-school research groups. The 
enthusiasm and accessibility of the Research Champion was seen as an important contributory factor. 


3. Dedicated time for research use 


One of the participating schools (in addition to the Research Champion’s own school) had ensured 
that its research lead had dedicated time to devote to research work. This school had made 
considerably more progress in terms of thinking and implementation than other schools. Notably it had 
been able to conduct its own investigations into questioning, an issue of particular relevance in this 
instance, and to introduce a new exercise book for children, in response to its findings. It was felt that 
this progress had been facilitated by the lead teacher’s ability to dedicate time to develop the project, 
and to engage with evidence in-depth. 


4. Simplicity of the approach 
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It was felt that approaches that were the ‘easiest to implement’ were often the most effective. One 
teacher (who was not one of the research leads) appreciated this practical approach—the application 
of research in the form of easily-applied real-world solutions that brought fast results: 


‘The research groups that we’ve had [have] really focused on doing things that are possible, 
the quick wins... so rather than spending a month planning one lesson, little quick fixes that 
we can get quick gains on. And if something doesn’t look like it’s working, then we'll put it on 
the backburner. So it’s really been a focus on practical, tangible things we can do’ (classroom 
teacher). 


Barriers and challenges affecting programme feasibility 


Research leads and other teachers were asked to reflect on the barriers they encountered—both in 
terms of making full use of the programme, and in using research evidence in teaching practice. The 
greatest challenge raised was one of competing time pressures: research leads—particularly those 
involved in preparing students for GCSE and A-level exams—felt that they would have liked, but were 
unable, to spend more time exploring research literature. Time pressures also made attending 
symposia or twilight events and scheduling research brokerage meetings challenging. 


‘Often there were just issues around cover... It was nothing to do with the programme itself; 
it’s just demands on time in my school’ (research lead). 


One research lead in particular felt that a lack of practical support from the school in the form of lesson 
cover and release time meant they were unable to engage with evidence—and the project more 
generally. While the school was not disinterested in the programme, it was felt that the senior 
leadership team faced other competing and more pressing issues meaning that the project lacked 
institutional support. Commitment from the school’s senior leadership team was seen as an important 
factor in the successful implementation, particularly to obtain school-level impacts. 


Another challenge identified by teachers was the ability to find a way to adapt and apply theoretical 
research evidence in the classroom. As the Research Champion noted: ‘we want to know what to do 
when we get into the classroom on Monday morning’. This was underpinned by a deeper confusion 
over ‘what research use constitutes’. 


Feasibility: Summary 


The programme largely ran as intended and as documented in the logic model in terms of the 
structure of outputs, but not in terms of participants’ engagement, so overall, the evidence on the 
programme's feasibility is mixed. The symposia were well attended and well received, with teachers 
valuing the opportunity to reflect on their own practice and to build networks. Attendance at the twilight 
forums, however, was not as good as intended, nevertheless most participating teachers felt that 
these were a useful and positive addition to the programme. 


At the heart of the project was the research brokerage service. While qualitative interviews with 
teachers revealed many positive elements to this service—most notably the accessibility and 
helpfulness of the Research Champion —survey feedback suggested that the wider teacher response 
to this element of the project was more mixed. This may have been connected to issues relating to 
time pressure (indeed, time pressures posed the most serious threat to the feasibility of the 
programme), but also concerned competing priorities in schools and varying levels of buy-in from 
senior leadership teams. Key issues, therefore, that must be considered in future implementations are 
the provision of adequate time to attend programme sessions, teaching cover for events, and buy-in 
from senior leaders. 


Perceived impacts 


Teachers taking part in qualitative interviews reflected on the perceived impacts of the pilot on their 
use of research evidence in teaching practice and pupil outcomes. These are discussed in turn. 
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The aim of the ATA Research Champion programme, as set out in the logic model, was to develop 
teaching expertise and practice by promoting educational research and the use of research in 
decision-making and educational practice—that improving teachers’ ability to access, question, and 
understand evidence would result in the implementation of lessons from research in their classrooms. 


As discussed in the implementation section, the extent to which this aim was achieved varied 
considerably from school to school. However, among teachers’ accounts were clear examples of 
research evidence being accessed and successfully applied. For example, one research staff group 
member (not a research lead) reported that he had changed the way he presented information to 
students: 


‘Asking students to constantly be doing high-level thinking all the time is a no go, in that they'll 
get exhausted very quickly, so I’ve started planning my lessons now with some easier ideas to 
start with and then every now and again give them a challenging idea, but for a short time 
period’ (classroom teacher). 


This teacher reflected that his ability to apply evidence in practice had been facilitated by the research 
group. It was felt that having the dedicated time for discussion and reflection in a supportive 
environment contributed to the success of this approach. 


In another school, the research lead had been given dedicated time out of the classroom to spend on 
the project. This time had been used to read and absorb evidence, and to understand how and why 
different approaches worked. The teacher had used this knowledge to conduct research with her 
pupils and as a result of her findings had overseen a number of changes in practice, including a new 
approach to questioning, and a new exercise book which allowed teachers to see the children’s 
workings. 


Even in those schools where there was little evidence of changed practice based on qualitative 
interviews, teachers felt they had been encouraged to reflect on their own approach to teaching, and 
had gained a better understanding of the research process. Much of this ability to engage with 
research was the result of having dedicated time (if only at symposium events) and the support of the 
Research Champion. 


‘I’m not an academic person... so to be inspired to read and to listen to research that has 
been done and wanting to do more... they've done really well to capture my imagination’ 
(classroom teacher). 


There was also evidence from the qualitative interviews that research leads had been empowered and 
enthused by the programme to independently seek out research evidence beyond that disseminated 
as part of the programme. For example, one research lead felt that increased awareness of evidence, 
coupled with dedicated time, had provided both the impetus and opportunity to read in-depth and fully 
understand how and why certain teaching approaches worked. Additionally, they felt that being able to 
learn something new, with a purpose and end point in mind, made the experience particularly 
valuable. Survey results, however, suggested that this may not have occurred more generally. 


In addition to this raised awareness in terms of research, one senior leader felt that the programme 
had had a wider impact on his staff, encouraging a new level of engagement with the profession. It 
was thought that this might lead to greater retention of NQTs. In addition, teachers felt that impacts on 
professional development, prompted by the programme, would have a lasting effect. 
‘Even if there’s nothing tangible... What it has done is engage [the teacher], professionalised 
[them], made [them] think about [their] own practice and, I'd even say that if [they'd] done 
nothing different, what that’s done on some level is make [them] a more engaged 
professional, and that will be positive in itself’ (research lead). 


Perceived impact on pupils 
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The programme did not set out with explicit short-term aims for impacts on pupils, however in the 
medium term (one to two years from the start of the intervention) the expectation was that the 
intervention would improve learning (Studying) skills, improve learning experiences, and improve 
educational attainment among pupils. It was anticipated that improvements in educational attainment 
would be sustained in the longer term. 


As with effects in other areas, the extent to which teachers felt the programme had impacted pupils 
varied considerably by school. Teachers noted that the programme was not at a point at which they 
would expect to see changes in pupil behaviour, that it was just ‘too early to talk about impacts on 
students’. In schools where there had been very slow progress and little change in teaching practice, 
teachers did not expect, or highlight, any changes in their pupils. However, in schools where the 
programme was more developed, teachers felt they had seen small changes in pupils which could be 
attributable to the programme. The effects on pupils were specific to the type of changes in practice 
schools had implemented. Examples include: 


1. Alternative provision*’ 


One of the participating schools had put alternative educational provision in place for any student not 
in mainstream lessons. (The school also operates a ‘no permanent exclusion’ policy.) As well as 
teaching, staff had a policy of offering tea and toast to alternative provision pupils whenever they felt it 
was necessary, and offering them pastoral support. While it was felt to be too early to talk about 
impacts on students, there was some anecdotal evidence that pupils’ attendance had improved, and 
that they found the alternative provision centre a safe and secure place to be (for some it was felt to 
offer a safe haven away from the difficulties of home life), and that this fostered engagement. The 
creation of a positive learning environment was seen as an essential first steps towards impacting 
attainment. 


2. Questioning and fostering a supportive learning culture 


A school which began with a focus on questioning—and went on to introduce changes to support 
children to share their workings with teachers, and combat the fear pupils had of making mistakes— 
felt that the programme had started to have a positive impact on the pupils’ attitude to learning. In 
particular, the way in which the changes encouraged a ‘have a go’ attitude and took the emphasis 
away from getting the correct answers to making effort. Establishing this culture was seen as an 
important step in creating an environment in which pupils could value the advantages of trial and error 
and not be afraid of getting things wrong. 


Despite general positive effects, one teacher felt that some pupils had been more resistant to change, 
and were reluctant to present unpolished work to their teachers. It was thought that high-achieving 
pupils, and those from certain cultural backgrounds, were less likely to have shown a response to this 
particular change in teaching practice. 


3. Feedback and assessment 


In the school which had implemented strategies to improve feedback, teachers also spoke positively of 
improvements, with pupils more able to identify where they needed to improve and what their next 
steps were. Reflecting on the impacts of this changed approach to feedback on students, one teacher 
said: 


*” Alternative provision is defined by the Department for Education as ‘education arranged by local authorities for 
pupils who, because of exclusion, illness or other reasons, would not otherwise receive suitable education; 
education arranged by schools for pupils on a fixed period exclusion; and pupils being directed by schools to off- 
site provision to improve their behaviour’. 
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‘| would say it’s effective... The students don’t always like to be forced to do questions, 
obviously, but | think if you asked them at the end of the topic if they understand it better, | 
think they do. | certainly think it’s effective in terms of improvement, obviously that’s the whole 
point of assessment, isn’t it, that they improve off the back of it. So |! would say they’re doing 
better because of that’ (classroom teacher). 


One teacher felt that changes to feedback implemented as part of the programme had had a positive 
effect on pupil confidence, independence, and self-efficacy. Changes meant that pupils took greater 
responsibility for their own learning, but were also proactive about seeking advice and comments from 
their teacher. It was thought that the new practices were becoming embedded at the school, changing 
pupils’ expectations. 


While it was generally accepted that the programme was at too early a stage to have impacted pupil 
attainment (a medium term impact for the programme), one teacher felt that students had started to 
develop a better understanding of the topics under discussion in his lessons, and that this would lead 
to an effect on their attainment. 


Readiness for trial 


Findings suggest that the Ashford Teaching Alliance Research Champion model is not ready to be 
evaluated in a trial in its current state of development. The main reasons for this conclusion include 
the mixed evidence in terms of programme feasibility, a lack of evidence of effectiveness, and an 
insufficient specification of the intervention in its current state of development. 


One key factor that hindered the feasibility of the programme was variation in the use of the support 
on offer (the research brokerage); in addition, there was variation in the engagement of teachers in 
terms of attendance at the programme events, and in the practical implementation of new knowledge 
in classroom practice. For example, although schools were asked to commit to releasing teachers for 
a certain amount of time to the project—for example to attend twilight forums—this was not something 
that happened in practice for all schools. 


Ensuring buy-in and engagement from senior leaders is of critical importance where an intervention 
aims to achieve a whole-school effect, so practical challenges identified in the process evaluation 
need to be addressed. These include difficulties presented by the changing structure and leadership of 
schools (which led one of the pilot schools to leave the programme); it would also be beneficial to think 
through the mechanisms for teachers directly involved in the programme to effectively cascade their 
newly obtained knowledge to colleagues. 


Most importantly, the core characteristics of the programme need to specified more precisely. This 
could be challenging for a programme designed to respond to the needs and interests of teachers and 
schools where a considerable level of flexibility in terms of content is an essential requirement. If the 
effectiveness of the programme relates to the structure of activities rather than content, then it would 
be useful to specify the minimum dosage for the programme to have an impact (for example, how 
many training sessions each participant should attend), and to specify how any new knowledge gained 
should be implemented in classroom practice. 


For a trial, pupil-level outcome data would need to be collected: this was not part of the pilot project. 
This would require careful consideration of (a) the year groups that should take part (a particular 
difficulty for a project which includes both secondary and primary schools), (b) which types of data 
would be appropriate as outcome measures (for example, attainment data only or attitudinal data as 
well), and (c) the timing of outcome data collection (so that data collection is in line with the logic 
model)—longer-term outcomes being measured a number of years after the start of the programme. 


Finally, given that a significant focus of the programme is on changing teachers’ perceptions and use 
of research evidence as a first step towards improving pupil attainment, it would be helpful if the trial 


collected outcome data at teacher level as well as at pupil level (with data collection being timed in line 
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with the logic model). Outcomes at teacher level should fully reflect the logic model, which was not the 
case in the pilot project where outcomes were developed by an external organisation and were not 
specifically tailored to the Ashford Teaching Allowance Research Champion project.” 


Cost 


Our estimate of the total cost of the intervention includes up-front costs, costs of providing the training 
and support, and costs of teachers attending the intervention events. 


Up-front costs 


There were costs associated with the preparation of the intervention before the implementation 
started. These included a launch meeting, staff cover costs for teachers attending the launch meeting 
(three teachers on average for each of the five schools), delivery of material, the Research 
Champion's salary (including costs associated with National Insurance, pension contributions, and 
overheads) and cover costs for the Research Champion—the latter are included in addition to salary 
costs in the intervention year as preparations for the intervention started in the final term of the 
previous academic year. 


The Research Champion used 60% of their time (as a full time teacher) in the four months before the 
programme started on preparing the intervention. This cost £9,875 plus a cover cost of £4,800. See 
Table 9 for more details. 


Table 9: Up-front costs 


Item Ol vit=m (ef: \Vcoeue ie F: \V a Oo) 629) Cost (£) per 
= 5 hrs) 


salt cover gunien meeting) 


pupil 


Research Champion (salary for 4 9,875 2.04 
months) 

Research Champion (cover for 4 4,800 

months) 


Sub-total up-front costs 15,845 


Base: 4,851 pupils receive the effect of the intervention per year. 


Cost of implementing the intervention 


During the implementation of the intervention, the majority of the costs were associated with hiring the 
venue (including refreshments) and speaker fees. There were a total of six speakers, each charging a 
standard fee of £600. Most of the speakers were based relatively locally and did not need 
accommodation. Accommodation costs were estimated on the basis of two nights of hotel 
accommodation, at £80 per night. There were no speaker fees associated with the twilight forums as 
these were run by the Research Champion and the cost was covered under their salary. Research 
Champion's travel costs associated with twilight forums and other school visits were around £200 in 
total. 


*® The outcomes survey was not tailored to the Ashford model in order to make the results comparable across all 
EEF projects in the ‘research use’ round. 


Education Endowment Foundation 33 


Ashford Teaching Alliance Research Champion 


The salary cost for the Research Champion was estimated as 60% of a full time teacher’s salary for 
12 months. This corresponds to £29,625. In addition, the Ashford Teacher Alliance employs an 
administrative assistant, and it has been estimated that work relating to this project accounted for a 
quarter of their time, costing around £4,000. 


The cost of purchasing (or printing and photocopying) resources and materials associated with the 
symposia and twilight forums was estimated as £1,520 for the year. 


In addition to the costs detailed above, schools incurred time costs associated with the research lead 
disseminating information related to the programme in their school. This cost was not translated into 
financial expenditure as it was planned ahead, which allowed the accommodation of lead teacher 
time. Therefore, we have indicated the time requirements in the table below but have not included any 
associated costs. Time spent by each research lead varied considerably from school to school but on average it 
was around half a day (2.5 hours) per half-term. See Table 10 for more details on the cost of providing the 
intervention. 


Table 10: Cost of providing the intervention (financial marginal cost) 


Item Time (days,1 day Cost (£) 
= 5 hrs) 


Speaker fees (£600 x 6 speakers) ee 
Research Champion salary (12 29,625 
months) 


Travel costs Research 
Champion 

Speakers’ accommodation (£80 
X 2 nights) 


Symposia venue & refreshments 

(eae aeympee) 
Twilights forum refreshments 
(oxeseion) 
| 
—— | 


Dissemination (5 research leads 
x half day per half term x 6 half 
terms) 


Sub-total cost of providing the 40,315 
intervention 


Base: 4,851 pupils receive the effect of the intervention per year. 


Cost and time of attending the intervention events 


The intervention events included three full-day symposia, six 90-minute twilight forums, and research 
brokerage visits of the Research Champion to the participating schools every half term. 


The symposia and the twilight forums required travelling to the venue, but because all schools were 
within 25 miles of each other, travel costs were fairly low. Teachers tended to share cars to go 
between the schools, so it was difficult to calculate precise travel costs. We estimate that travel 
expenses were around £10 per teacher per year. 


Each teacher was expected to use three days to attend the symposia. With forward planning, staff in 
every school were released for these events without cover being bought in. 
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Total staff time associated with attending the intervention sessions breaks down as follows: 
1. 3 staff members in each of the five schools attending all three full-day symposia (45 days in 
total); 


2. 3-staff members in each of the five schools attending all six 90-minute twilight forums (27 days 
in total); and 


3. 3 staff members in each of the five schools spending two days per half term at research 
brokerage meetings (180 days in total). 


Table 11 shows a breakdown of costs and time associated with attending the intervention training 
sessions. 
Table 11: Costs of attending the training sessions 


Item Time (days, 1 | Cost (£) Cost (£) per 
day = 5 hrs) rede} ey 


Teachers’ travel costs (£10 x 3 
150 
teachers x 5 schools) 


Sub-total costs of attending the 
sessions 
Base: 4,851 pupils receive the effect of the intervention per year. 


Total cost and cost per pupil 


This intervention was implemented at the whole-school level and intended to benefit pupils across all 
year groups. However, the implementation was gradual, with some pupils receiving the direct effect of 
the intervention only at the end of the academic year. Therefore, to calculate the total cost per pupil in 
the intervention’s first year, we considered only those pupils who were involved in the programme from 
the beginning of the academic year (n = 2,075). When estimating costs per pupil in the intervention’s 
second and third years, we assumed that all pupils at each school would be affected by the 
intervention from the beginning of the academic year, and thus used the total number of pupils across 
the five participating schools (n = 4,851).Table 12 shows the total cost of the intervention and its cost 
per pupil. Overall, the cost of the programme to schools was low. Over the first year of the intervention 
taking place, schools spent £27.13 per pupil. 


Total cost was divided between start-up costs and running costs. The launch event, for example, may 
be regarded by schools as a one-off expenditure which will not apply in subsequent years if the 
intervention is extended for a longer period of time. This separation is particularly important as it also 
allows us to estimate the potential cost of the intervention if it is implemented for three years, where 
we distribute the start-up costs over the whole length of the programme. 


Table 13 shows how the average cost per pupil per year decreases as the length of the interventions 
is increased. Additionally, since the number of pupils receiving the intervention is estimated to 
increase after the first year; there is an important reduction of costs per pupil over time. 


Table 12: Total cost and cost per pupil in the first year 
Item Cost (£) | Cost (£) per pupil 
8 


‘Startupcost | 15845 


Running costs per year 40,465 
Total cost - first year 56,310 
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Table 13: Total cost per pupil over three years of the programme 


Item Cumulative cost per pupil (£) Average Cost (£) per pupil per year 


27 27 
‘Year? 


Year2 
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Conclusions 


Formative findings 


Findings from depth interviews suggest that the key learning points to take forward for the ATA 
Research Champion project include: 


1. exploring ways to ensure participating staff are given regular, dedicated time for the 
programme, in particular release time to attend all events and to engage with the brokerage 
service, and time to plan, implement, and review changes in classroom practice; 


2. fostering support from senior leaders at the school: encouraging buy-in from senior leadership 
teams will lead to more support for staff, including release time and classroom cover, as well 
as a greater likelihood that learning from the project will be shared and taken forward across 
the whole school; 


3. allowing flexibility for schools to tailor the strategies they use to their school context—viewed 
as key to promoting engagement and buy-in from teachers and senior leadership teams; and 


4. providing practical examples and materials that can be used in the classroom to facilitate 
classroom implementation, with a focus on simple strategies expected to bring ‘quick gains’. 


Limitations of the evaluation 


There were a number of limitations to our evaluation of the ATA Research Champion programme: 


1. The outcomes collected in the teachers’ survey were not wholly consistent with those outlined 
in the intervention’s logic model. 


2. There was no comparison group. 


3. Response rates to the teachers’ surveys (63% at baseline and 56% at follow-up) mean we 
were not able to capture data on all teachers at the participating schools. It is possible that 
there was a systematic bias associated with non-response (for example, responses may have 
come from teachers who had more positive attitudes towards research, or who were less 
busy, than those who did not respond). 


4. The sample size available for analysis of outcomes (n = 106) was very low, and the 
differences would need to be very large in order to be statistically significant. 


5. It was not possible to undertake a separate analysis of outcomes for teachers with direct 
involvement in the programme due to low sample size in that group. 


6. No pupil-level data was collected as part of the evaluation. 
Interpretation 


The ATA Research Champion evaluation aimed to explore whether, and to what extent, research 
communication and engagement strategies employed by the intervention had the potential to improve 
teachers’ use of, and attitudes towards, academic research and thus support pupils’ progress. 
Analysis of data from the survey of teachers showed that there had been no significant changes in the 
teachers’ attitudes towards research, or their actual use of research evidence in practice between 
baseline and follow-up; there was, in other words, no evidence to show that the intervention had 
achieved its aims in its pilot year. 


The programme was largely delivered as intended and as documented in the logic model, however 
while the structure of the programme and its outputs were as planned, levels of participant 
engagement varied. Therefore, overall, the evidence on the programme’s feasibility was mixed. 


In terms of the individual components of the programme, the research symposia were well received by 
participants who valued the opportunity to learn about developments in educational research and 
reflect on teaching practice outside the classroom. Likewise, the twilight forums, although not as well 
attended as anticipated, were generally felt to be a useful and positive addition to the programme. 
Evidence on the research brokerage service and ad hoc support was more mixed: while many 
participants valued this element of the intervention, in particular the accessibility and helpfulness of the 
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Research Champion, others did not find it as useful. This may have been related to the barriers that 
some teachers faced finding time to fully engage in the programme, which was related, in turn, to 
competing priorities in schools and varying levels of buy-in from senior leadership teams. Indeed, time 
pressures posed the most serious threat to the feasibility of the programme. Overall, findings suggest 
that systematic implementation of research evidence would require an enhanced process of mentoring 
and support to promote sustained engagement at the senior leadership level. 


Given the mixed evidence in terms of the programme feasibility and a lack of evidence that the 
programme changed teachers’ attitudes towards research evidence and their actual use of research in 
their practice, our conclusion is that the ATA Research Champion programme is not ready to be 
evaluated in a trial in its current state of development. A number of development points need to be 
considered before the intervention could be taken to a full trial stage: 


1. Given the ‘bespoke’ nature of the programme—aiming as it does to respond to needs of 
individual participating schools—what are its defining features? For example, how important is 
it to follow a specified structure (such that, for example, teachers attend all symposia and 
twilight forums), and how important are the topics covered at the events? Before going to trial, 
it would be necessary to specify exactly which components are required, and how much 
commitment would be needed from teachers (that is, the minimum dosage for the programme 
to be effective). 


2. The issues with implementation (time pressures on teachers, staff turnover, and buy-in from 
senior leadership) would need to be fully addressed before going to trial. For example, 
although schools were asked to commit to releasing teachers for a certain amount of time to 
the project (so they could, for example, attend the twilight forums) this was not something that 
happened in practice for all schools. Better strategies are needed for tackling barriers to 
successful implementation within schools. 


3. For a trial, pupil-level outcome data would need to be collected; this was not part of the 
evaluation of the pilot. It will require careful consideration of the year groups that should take 
part—a particular challenge for a project which includes both secondary and primary schools. 


4. The programme needs to consider fully what types of data would be appropriate as outcome 
measures at pupil level—for example, attainment data only, or attitudinal data as well, 
including educational aspirations as a longer-term outcome (see the logic model). 


5. The timing of outcome data collection needs to be in line with the logic model. It is likely that a 
project of this kind will require pupil attainment data to be collected some time after the end of 
the intervention. 


6. Given that a significant focus of the programme is on improving teachers’ attitudes and 
behaviours as a first step towards improving pupil attainment, it would be helpful if the trial 
collected outcome data at the teacher level as well as at the pupil level (with data collection 
being timed in line with the logic model). Outcomes at the teacher level should fully reflect the 
logic model. 


7. More conceptual thinking should be given as to how the programme can create a whole- 
school effect when only a small number of teachers have direct involvement in the 
programme. In particular, it needs to be made more explicit whether and how teachers directly 
involved in programme can effectively cascade their newly obtained knowledge to colleagues. 


Future research 


Our findings suggest that the ATA Research Champion programme is not yet ready to be evaluated in 
a trial and needs further development before additional research is conducted to explore its 
effectiveness. As this programme was funded and evaluated as part of the EEF’s ‘research use’ 
round, it may be useful to review findings from all projects together. This could help to draw out key 
lessons about which strategies prove particularly successful when encouraging teachers to engage 
with research evidence. This learning could usefully be fed into the Research Champion programme 
as it develops. 
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Appendix A: Outcome measures 


The five outcomes measures used in the analysis were developed by Poet et a/. (unpublished). 
Positive disposition to academic research in informing teaching practice 


Composite scale measures constructed using components of different survey questions were used to 
compare before and after measures. The first of these measures combined responses to six items: 


4 Ease of understanding reports, books, or summaries based on academic research (Q6, item 
1). 


2. Information from research plays an important role in informing my/our teaching practice (Q8, 
item 1). 


3. | know where to find relevant research that may help to inform teaching methods/ practice 
(Q8, item 3). 


4. |am able to relate information from research to my context (Q8, item 5). 
| feel confident about analysing information from research (Q8, item 7). 


6. | use information from research to help me decide how to implement new approaches in the 
classroom (Q8, item 9). 


The Cronbach's alpha for measure 1 at both the baseline and outcome survey was 0.86 indicating that 
the questions were highly correlated. 


Use of academic research to inform selection of teaching approaches 


The second composite measure was constructed by combining four criteria: 


1. Articles, reports, books or summaries based on academic research (paper or web based) 
were important when identifying the approach (Q3b, item 4). 


2. Online evidence platforms or databases (such as the Sutton Trust Teaching and Learning 
Toolkit) were important when identifying the approach (Q3b, item 9). 


3. ‘Influence the fact’ approach was backed by academic research had on decision to adopt it 
(Q4, item 6). 


4. The extent to which articles, reports, books, or summaries based on academic research 
(paper or web based) are consulted when deciding on approaches to support pupil’s progress 
(Q5, item 3). 


This deviates from the NFER guidelines that suggest the outcome measure is constructed with the 
questions above in addition to the statement: ‘CPD based on academic research was important when 
identifying approach’ in Q3a. Since this latter statement is only included in the outcomes survey, using 
the NFER survey and analysis guidelines it would only have been possible to assess ‘use of academic 
research to inform selection of teaching approaches’ in the outcomes survey. However, by excluding 
this particular question from the analysis it was possible to compare results across the two time points. 
The Cronbach’s alpha for the measure was 0.37 at baseline and 0.54 in the outcomes survey, 
indicating a weak internal consistency at baseline and a moderate one in the outcomes survey. 


*® Teachers who responded ‘I don't use this source’ to question 6, item 3, have been assigned a score of 2.5 on this measure, 
as have those with missing values. This coding follows the evaluation guidelines set out in Poet et a/. (unpublished). 
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Perception that academic research is not useful to teaching 


Measure three captured teachers’ ‘perception that academic research is not useful to teaching’ 
combining responses to two survey questions: 


1. | do not believe that using information from research will help to improve pupil outcomes (Q8, 
item 2). 


2. Information from research conducted elsewhere is of limited value of our school (Q8, item 8). 


The Cronbach’s alpha for this measure was 0.60 at baseline and 0.72 in the outcomes survey, 
suggesting a moderate level of correlation between measures. 


Perception that own school does not encourage use of academic research 


The fourth outcome measure aimed to capture the ‘perception that the teachers’ own school does not 
encourage use of academic research’, combining responses to two items on question 8: 


1. My school leaders/governors do not encourage me to use information from research to 
improve my practice (item 4). 


2. Other staff in my school rarely use information from research to inform their teaching practice 
(item 6). 


The Cronbach’s alpha for this outcome measure was lower than others, 0.71 at baseline and 0.37 in 
the outcomes survey, suggesting a weaker correlation between measures in the outcome survey, and 
therefore lower reliability. For this reason, this composite measure was not considered robust enough 
to be included in the analysis and was excluded from the report. Instead the two questions were 
analysed separately. 


Active engagement with online evidence platforms 


The measure constructed to explore ‘active engagement with online evidence platforms’ combined 
questions exploring the extent to which teachers used online evidence platforms, such as the Sutton 
Trust Teaching and Learning Toolkit (Q5, item 6) and how easily they understood them (Q6, item 3) aa 
The Cronbach’s alpha for the measure was 0.66 at baseline and 0.61 in the outcomes survey, 
showing a moderate correlation between the two measures. 


© Teachers who responded ‘I don’t use this source’ to question 6, item 6, have been assigned a score of 2.5 on this measure, 
as have those with missing data. This coding follows the evaluation guidelines set out in Poet et al. (unpublished). 


Education Endowment Foundation 41 


Ashford Teaching Alliance Research Champion 


Appendix B: Memorandum of Agreement 


A 
NatCen AIP 


Social Research that works for society 


Agreement to participate in the Evaluation of Ashford Teaching Alliance: 
Research Champion Project 


Please sign both copies, retaining one and returning the second copy to Ben White at Highworth 
Grammar School, Ashford. 


School Name: 
Aims of the Evaluation 


The aim of this project is to evaluate the impact of the ATAResearch Champion project, 

a scheme intendedto promote effective use of educational research to raise educational 
outcomes by developing teaching expertis eand practice. Theresults of theres earch will 
contribute to our understandingof what works in raisingthe pupil's attainment and will be widely 
disseminatedto schodsin England. Ultimately we hope that the evaluation will equip school staff 
with the skills and understanding to better support children throughmaking effective use of high 
quality educational research in key decisions which they make. 


What will the Research Champion (RC) Projectprovide? 
The Research Champion project will provide the following free of charge to each school: 


1. Three Research Symposia: These are staff development days based onthe 
development aims of thesix partnerschools. They will incude keynotes eminars from 
educational researchers and workshops in which their key ideas will be exploredin more 
depth. 

2. Termly Research and Development twilightforums: Thesewill providea context to 
explorehow the use of research both in classrooms and at school leaderlevel. 

3. Bespoke research brokerage for each individual school: Each school involved can 
make use ofthe RC in a ‘research brokering role once aterm. This couldinvolve 
providing a summary of relevant res earch findings, running a workshop with a group of 
staff, or providing your SLT with findings relevant to your development plans. 


Structure of the Evaluation 


The evaluation is being conducted by NatCen. NatCenwill cometo visitschools once or twice 
during the programme to see how the Research Champion model is working in each school, We 
will talk to teachers to find out how thetrial is working for you, and the mechanisms for this. 
Telephone cor ersations will be scheduled between visits. This level of evaluation is expected to 
be the same in all 6 schools participating in the trial. 


In September 2014 and May-June 2015 NatCen will carry out a short online survey with all 
teaching staffin participating schools. Analysis of anonymised National Pupil Database data, 
held by the Department for Education, will be used to help understand whether the programme 
has an impact on Key Stage 2, 3 and 4 attainment. 


Use of Data 


Data collected regarding teachers and pupils will be treated with the strictest confidence. No 
individual school, teacher or pupil will be identified in any report arising from the research. 
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A 
NatCGen AIP 


Social Research that works for society 


2 ibiliti 
ATA and the Research Championwill: 


© Deliver the three symposia, sixtwilight sessions, and bespoke ‘brokerage’ sessions 
outlined above. 

® Bethe first point of contact for any questions about the evaluation 

e Provide on-going support to the school 

* Send out regular updates on the progress of the project through a newsletter 


NatCen will: 


® Correspond with schods to evaluate the project including visits, observations, face to 
face and telephone interviews with staff, collecting written feedback and policy/quidance 
documents and surveying teaching Staff in the school 

* Collect and analyse data from the project 


* Disseminate research findings 


The School will: 


® Nominate a ‘researchlead whowill be the primary cortact point between the schooland 
both the research champion and evaluation team. 

e Liaise with Ben Whiteto complete aninitial researchinterests audit in June/July 2014. 

e Release Staff sothat they can attend training sessions (outlined below) and take partin 
up to three interviews with NatCen 
- At least three staff to attend each Research Symposium 
- At least two staff to attend each twilight session. 

® Committo utilising the ‘research brokerage’ support which can be tailored to meet 
schools’ individual needs and interests. 

© Provide alist of allteaching staff tothe project team, to be shared with the evaluation 
team 

e Allow time foreach survey phase and encourage staff to participate in project surveys 
and interviews 

e Ensure the shared understanding and support of all school staff for to the project and 
personnel involved 

e Share school policy and guidance documents with NatCen 


We commit to the evaluation of the ATA Research Champion as detailed above 


Head teacher. 
Research Lead: 
Date: 
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Appendix C: Letter to headteachers 


Name 
School name 
Address 


Date 


Dear name, 


The Evaluation of Ashford Teaching Alliance: Research 
Champion Project 


| am writing to you to let you know that over the next few week’s teachers in your school 
will be taking part in an online survey for the Evaluation of Ashford Teaching Alliance: 
Research Champion Project. 


This survey will help us to understand teachers’ views and experiences of using research in 
their teaching. The study is funded by the Education Endowment Foundation. All information 
collected in this survey will be confidential and individuals will not be identified in the 
published results. 


We would appreciate your support in encouraging all teachers to complete the survey. 


Further information about the study can be found at www.natcen.ac.uk/research-use. If 
you have any queries, please feel free to email researchuse@natcen.ac.uk or call us on 0800 
652 0401. 


J Speipf— 


Dr Svetlana Speight 
Research Project Director 


NatCen Social Research 
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Appendix D: Survey invitation email to teachers 


[Serial] 


To [firstname lastname], 


| am writing to invite you to take part in a short piece of research for the Evaluation of Ashford 
Teaching Alliance: Research Champion Project. 


As you may be aware, as part of this programme, teachers in your school will be involved in learning 
how they may be able to enhance your school’s teaching using research-based evidence. 


As part of this process we would like you to answer a few questions about your own teaching practice. 
The online questionnaire should take no longer than 15 minutes. We would be grateful if you could 
complete it by [date]. 


Take part now 


The information you provide will help us to understand teachers’ views and experiences of using 
research in their teaching. The study is funded by the Education Endowment Foundation. All 
information collected in this survey will be confidential and individuals will not be identified in the 
published results. 


Further information about the study can be found at http://www.natcen.ac.uk/research-use. If you 
have any queries, please feel free to email researchuse@natcen.ac.uk or call us on 0800 652 
XXXX. 


Kind regards, 


Dr Svetlana Speight 
Research Project Director 


NatCen Social Research 
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Appendix E: The Outcomes Survey 


Supporting Pupil Progress 


Thank you very much for taking part in this survey. Your responses will contribute to a study 
conducted by NatCen Social Research on behalf of the Education Endowment Foundation. It is 
exploring different approaches to improve pupil progress. The survey includes questions on how 
you/your school have decided to introduce new approaches and the types of information you use to 
inform decisions on teaching and learning. 


The survey should take no more than 15 minutes to complete. 


Your answers will be treated confidentially, which means that you and your school will not be identified 
in any reports produced from this research. 


BLOCK A - Introduction 
Job 


1. What is your job role? Please tick one box below that best describes your role 


Classroom teacher 


Middle leader (e.g. head of department, subject or curriculum area leader, key 
stage leader, pastoral services leader) 


Senior leader (e.g. deputy or assistant headteacher) 


Headteacher, principal or director 


OO ee OO 


Other role (please specify ) 


HowLong2 
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2. How long have you been in the teaching profession? (Please tick the box that describes 
the length of your whole teaching career, including career breaks) 


30 years ormore [] ‘ 5-9 years [] 4 
20-29 years LJ 2 1-4 years L] 5 
10-19 years L] 3; First year of teaching (NQT) L] «¢ 
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BLOCK B - About a specific approach to supporting pupils’ progress 


ApproachO 


Please name in the box below a specific approach that you have used within the last two 
years to support pupils’ progress. For example this could be a teaching method, or a 
3a. resource, product or initiative. 


Name/brief description (Please write in the box below) 


Identify3 


3b. Which, if any, of the following were important in identifying the approach you named? 
Please select the three most important sources. (Please tick up to three) 


Ideas generated by me or my school 
Ideas from other schools 
Advice from my local authority or academy chain 


Articles, reports, books or summaries based on academic research (paper or web 
based) 


Articles, reports, books or summaries based on teacher experience (paper or web 
based) 


The promotional materials of an external supplier 


Oi O Bee ee 


Action research conducted by me or my colleagues 
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Information gathered through training/CPD 


Online evidence platforms or databases (e.g. the Sutton Trust Teaching and 
Learning Toolkit) 


Guidance from official bodies such as DfE and Ofsted 
Guidance from exam boards 


Don’t know 


Oa Oe Ll ee 


Other (please specify) 13 


eee 
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CPD 


Please indicate, from the list below, what the training/CPD was based on. (Please tick all 


that apply) 
3c. 
Exam board information [] ‘: 
Academic research [] , 
Ideas from my school (e.g. internal INSET) [] 3 
Ideas from other schools [] 4 
Expertise of an external consultant [] 5 
Expertise of a programme provider L] 6 
Local authority/academy chain guidance L] 7 
Other (please specify) LI] , 
Influence 
Please rate the level of influence that each of the following factors had on the decision to 
adopt your approach. (Please tick one box in each row) 
Strong Some No Not 
We thought the approach... influence influence influence applicable 
1 2 3 4 
...would be straightforward to implement [| [] [] [] 
...was likely to be popular with staff C] [| C] C] 
...was likely to be popular with parents [| [| [| [] 
...was likely to be popular with pupils L] [I L] L] 
... Was inexpensive [| | | | 
...was backed by academic research [| | | | 
...Was a good fit with existing practices | | | | 
...aligned with our professional experience [] C] C] [| 


Education Endowment Foundation 50 


Ashford Teaching Alliance Research Champion 


BLOCK C - Your general approach to teaching and learning to support pupils’ progress 


We would now like you to think more broadly about how you develop your teaching to support pupils’ 
progress. 


5. ConsultPP 
To what extent do you consult the following sources when deciding on your approaches to 
support pupils’ progress? (Please tick one box in each row) 
A lot Alittle Notat all 
1 2 3 
Pupil performance data [| [| | 
External organisations (e.g. local authority, academy chain, DfE or Oo Oo Oo 
Ofsted) 
Articles, reports, books or summaries based on academic research g gO g 
(paper or web based) 
Articles, reports, books or summaries based on teacher experience Oo Oo Oo 
(paper or web based) 
Information gathered through training/CPD | im [| 
Online evidence platforms or databases (e.g. the Sutton Trust Oo O O 
Teaching and Learning Toolkit) 
Guidance from exam boards [| [| [| 
Colleagues within my own school [] | | 
Colleagues in other schools [| [| | 
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UnderstandPP 


6. How easy do you find it to understand the information that these sources provide about how 
to support pupils’ progress? (Please tick one box in each row) 


Very Quite Notvery  Notat all | don’t use 
easy easy easy easy this source 
1 2 3 4 5 
Pupil performance data [| [| [| [| [| 
External organisations (e.g. local authority, 
academy chain, DfE or Ofsted) = O O = O 
Articles, reports, books or summaries based 
on academic research (paper or web based) O O O O U 
Articles, reports, books or summaries based 
on teacher experience (paper or web based) CL] O O O O 
Information gathered through training/CPD [| [| | | [| 
Online evidence platforms or databases (e.g. 
the Sutton Trust Teaching and Learning [| [| | [| [| 
Toolkit) 
Guidance from exam boards [| | | | im 
Colleagues within my own school L] L] L] L] L] 
Colleagues in other schools | [| [] [] [| 
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BLOCK D Evidence-based teaching and using evidence from research 
Evidence3 


What does the term ‘evidence-based teaching’ mean to you? 


Please select up to three boxes that best describe your understanding of the term. 


Conducting action research and applying the learning 
Learning from colleagues and applying the learning 
Applying Ofsted or DfE guidance 


Using an online evidence platform/database (e.g. Sutton Trust Toolkit) and applying 
the learning 


Applying exam board guidance 


Using pupil performance data to track pupil progress and plan ahead 
Applying the recommendations of an external supplier 


Reading and applying information from academic research or from working with 
researchers 


10 


L 
al 
L 
s 
L 
Combining academic research evidence with my professional expertise [] 6 
L 
J 
LJ 
Learning from external consultants, trainers or advisors [_] 
LJ 


| don’t know 11 
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Research 


This question aims to find out how (if at all) you use research information in your work. By 
8. ‘research’ we mean information from books, reports, articles, summaries, training or events that 
is based on academic studies. 


Please indicate the extent to which you agree or disagree with the following statements. (Please 
tick one box in each row). 


Strongly Agree Neither Disagree Strongly 
agree agree nor disagree 
disagree 
3 
1 5 
es 4 
Information from research plays an important gO Og gO gO gO 
role in informing my/our teaching practice 
| do not believe that using information from 
research will help to improve pupil outcomes O O = UO O 
| know where to find relevant research that 
LO LO LO LO L 


may help to inform teaching methods/practice 


My school leaders/governors do not 
encourage me to use information from L] L] [| [| [| 
research to improve my practice 


| am able to relate information from research to 
my context U U U U U 

Other staff in my school rarely use information 
: 7 O O O O O 


from research to inform their teaching practice 


| feel confident about analysing information 
from research O UO O O O 


Information from research conducted 
elsewhere is of limited value to our school U O U O U 


| use information from research to help me to 


decide how to implement new approaches in [| [| | [| [| 
the classroom 
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ResUsed 


In the last year, how (if at all) have you used information from academic research to inform 
your practice? (Please tick all that apply) 


Respondents can either tick ‘| have not used information from research...’ OR as 
many responses as they like from items 2 to 7. 


Respondents are routed as follows: 


1. ‘have not used information from research’ (item 1) — go to Q11. 


2. EITHER or BOTH ‘change classroom practice’ (item 4) /influenced colleagues to 
change their practice (item 6), regardless of whether they have ticked any 
other options — go to O10 


| have not used information from academic research in the last year U 1 
Or, in the last year | have used information from academic research to: 

discuss best practice with colleagues a 2 
reflect on my own practice LI 3 

change classroom practice (this could be starting, developing or discontinuing an 
approach) [I 
contribute to my own research/enquiry = 5 
influence colleagues to change their classroom practice (this could be starting, (ale 

developing or discontinuing an approach) 

L 


improve my knowledge of a topic or subject 


{Select if ResUse = 4 and/or 6} 


Change 


10. What was it about the research information that enabled you to change classroom 
practice? (Please tick all that apply) 


It was clear (e.g. language, style, presentation) L] 1 
Itwas convincing L] 2 


| was able to discuss the research with a researcher or someone else who Oo 3 
understood it 
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| could see clearly how the research related to our context 


There was coaching and training available based on the research 
It contained practical guidance about how to apply the research in the classroom 


| was able to see the research being applied in another school 
It encouraged collaborative enquiry 


It was supported by resources (e.g. funding, materials) 


Iowa | he 


Other (please specify) 10 
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BLOCK E - Your knowledge about research 


In this section we would like to gather some information about your knowledge of research. Please 
answer the questions without referring to other sources. 


Knowledge 


11. Current understanding from academic research suggests that each of the following 
statements is ‘true’ or ‘false’. (Please tick the answer that you know to be correct in each row. 
If you are not sure, please tick ‘don’t know’). 


True False Don’t know 
The research says that: 
1 2 3 
Drinking six to eight glasses of water per day improves pupil gO O gO 
learning outcomes 
Reducing class size is one of the most cost-effective ways to 
: ‘ Oo Oo Oo 


improve pupil learning outcomes 


Extending the school day is more likely to improve learning 
outcomes for pupils on Free School Meals than pupils not on L] L] L] 
Free School Meals 


Interventions that focus solely on raising pupil aspirations have Oo Oo Oo 
little impact on learning outcomes 


Setting pupils by ability improves learning outcomes for all 
g pup y y imp g O Oo O 

pupils 

Individual pupils learn best when they receive information in 
pup y Oo Oo Oo 


their preferred learning style (e.g. auditory, visual, kinaesthetic) 
Peer tutoring (students supporting other students with their 

learning) usually benefits the pupil being tutored more than the L] L] L] 
pupil doing the tutoring 


Homework has a greater impact on pupils’ learning outcomes Oo i O 
at secondary school than at primary school 
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Methods 


12. Beloware descriptions of three reasons why someone would want to carry out 
research. Along the top of the table are five different research methods. 


Please match the research purpose with the best research method for achieving it by selecting 
the relevant option. Please select one box in each row. There are only three matches — two 
methods are incorrect (please do not use the same answer more than once). 


Randomised Longitudinal Interviews Literature Correlational 
Controlled study and/or review study 
Trial questionnaires 
1 3 
2 4 5 
To provide an overview of the 
evidence base 
L L] LJ L] L 
To determine whether an 
intervention or approach has a 
direct impact on pupil learning il L] Ey) L] L] 
outcomes 
To understand how an intervention 
or approach works in practice LJ L] LJ L] L] 
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BLOCK F - About the intervention 


The next questions relate to your schools participation in the ATA Research Champion programme 


You/your school has been invited to take part in the ATA Research Champion programme. The 
following set of questions asks about your views on this initiative. 


Participation 


13. Did your school take part in the Ashford Teaching Alliance: Research Champion 
Project? (Please tick one box only) 


Yes, and | had first-hand involvement [_] 


Yes, my colleague/s were involved, and they shared the learning [] 5 
with me 


Yes, my colleague/s were involved, but | don’t know any more L] 3 
about it 


No, my school did not take part Ll , 


I’m not sure L] 5 


{Select if Participation = 1&2} 


The following questions ask you to comment on your involvement in the Ashford Teaching Alliance: 
Research Champion Project. In these questions we would like you to think about your experiences 
of the whole project rather than its specific components. 


Information1 


14. How did you feel about the amount of information that was provided through the 
project? (Please tick one box only) 


There was too much information UL 1 
The amount of information was about right LI 


There was too little information L] 3 
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{Select if Participation = 1&2} 


Information2 


15. To what extent did the information from the project enable you to... (Please tick one box 


in each row) 


discuss best practice with colleagues in my school 


share the learning with people or organisations outside my 
school 


reflect on my own practice 


change classroom practice (this could be starting, 
developing or discontinuing an approach) 


reinforce existing practices 


conduct my own research or enquiry 


influence colleagues in my school to change _ their 
classroom practice (this could be starting, developing or 
discontinuing an approach) 


improve my knowledge of a topic or subject 


Education Endowment Foundation 


A lot 


O 


A little 


O 


Not at all 


oO 
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{Select if Participation = 1&2} 


ResultQ 
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16. To what extent do you agree or disagree with the following statements? (Please tick one 


box in each row) 


As a result of the project | have: 


Sought out further evidence, 
research or information about a 
topic relevant to my practice. 


Sought out further evidence, 
research or information on other 
topics/approaches. 


{Select if Participation = 1&2} 


Strongly 
agree 


LJ 


Agree 


Neither Disagree Strongly 
agree nor disagree 
disagree 


In the following question we would like you to think about the individual elements of the Ashford 
Teaching Alliance: Research Champion Project rather than the project as a whole. 


Elements 


17. How would you rate the following elements of the Ashford Teaching Alliance: Research 
Champion Project? (Please tick one box in each row) 


Very Quite 

good good 
Symposia L a 
Twilight sessions L L 
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Average 


Quite 
poor 


il 


Very Did not use/did 
poor not attend 
_| LI 
L 
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Brokerage (the 

Project Lead 

coming to your L LI L eal a | 
school and offering 

advice) 

Individual 

supportiadvice from 

the Research into LO L] L Ie LJ LJ 
Practice lead via 

phone/email 


{Select if Participation = 1&2} 


Recommend 


18a. Overall, would you recommend the Ashford Teaching Alliance: Research Champion 
Project to another school? (Please tick one box only) 


Yes, definitely L 1 
Yes, probably LI 2 
Probably not LI ; 


Definitely not | 4 


{Select if Participation = 1&2} 


Reason 


18b. Please explain your reasons: 
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Thank you very much for completing the survey. 
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You may re-use this document/publication (not including logos) free of charge in any format or 
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